An Individualized Risk Calculator for Research in Prodromal Psychosis
Publication: American Journal of Psychiatry
Abstract
Objective:
Approximately 20%–35% of individuals 12–35 years old who meet criteria for a prodromal risk syndrome convert to psychosis within 2 years. However, this estimate ignores the fact that clinical high-risk cases vary considerably in risk. The authors sought to create a risk calculator, based on profiles of risk indicators, that can ascertain the probability of conversion to psychosis in individual patients.
Method:
The study subjects were 596 clinical high-risk participants from the second phase of the North American Prodrome Longitudinal Study who were followed up to the time of conversion to psychosis or last contact (up to 2 years). The predictors examined were limited to those that are supported by previous studies and are readily obtainable in general clinical settings. Time-to-event regression was used to build a multivariate model predicting conversion, with internal validation using 1,000 bootstrap resamples.
Results:
The 2-year probability of conversion to psychosis was 16%. Higher levels of unusual thought content and suspiciousness, greater decline in social functioning, lower verbal learning and memory performance, slower speed of processing, and younger age at baseline each contributed to individual risk for psychosis. Stressful life events, trauma, and family history of schizophrenia were not significant predictors. The multivariate model achieved a concordance index of 0.71 and, as reported in an article by Carrión et al., published concurrently with this one, was validated in an independent external data set. The results are instantiated in a web-based risk prediction tool envisioned to be most useful in research protocols involving the psychosis prodrome.
Conclusions:
A risk calculator comparable in accuracy to those for cardiovascular disease and cancer is available to predict individualized conversion risks in newly ascertained clinical high-risk cases. Given that the risk calculator can be validly applied only for patients who screen positive on the Structured Clinical Interview for Psychosis Risk Syndromes, which requires training to administer, its most immediate uses will be in research on psychosis risk factors and in research-driven clinical (prevention) trials.
Given limitations of available treatments for schizophrenia, with most patients showing substantial deficits in social and occupational functioning throughout life, there is considerable interest in developing preventive approaches to psychotic disorders (1). Ascertainment of individuals at greatest risk is crucial to these efforts. For the majority of patients, onset of fully psychotic symptoms is preceded by the emergence of subtler changes in belief, thought, and perception that appear to represent attenuated forms of delusions, formal thought disorder, and hallucinations, respectively. Among individuals 12–35 years old with a recent onset of such symptoms (termed clinical high-risk cases), approximately 20%−35% develop fully psychotic symptoms over a 2-year period, an incidence rate that is more than 100 times larger than in the same age band in the general population (2). Furthermore, it appears that the clinical high-risk criteria are sensitive to an imminent risk for onset, as most of the conversions occur during the first year after ascertainment, with a decelerating conversion rate thereafter (3).
Although clinical high-risk criteria have been validated in epidemiological studies as sensitive to conversion risk, their utility in individual decision making is currently limited, given that 65%–80% of cases ascertained by these methods do not convert to psychosis within a 2-year time frame. About a dozen studies have examined combinations of clinical and demographic variables to determine whether prediction of psychosis can be enhanced beyond the 20%−35% risk associated with clinical high-risk status (4). Multivariate algorithms requiring particular combinations of symptoms and demographic factors achieve relatively high positive predictive values and specificity (e.g., in the 50%−70% range) but low sensitivity (e.g., in the 10%−30% range) (3). There is consistency among studies in showing (unsurprisingly) that greater severity of the psychosis-risk symptoms at baseline is the best predictor of conversion; nevertheless, the most predictive multivariate profiles vary across studies (as summarized in reference 4). Although it should be noted that few studies have attempted direct replication of one another’s risk algorithms, this pattern suggests heterogeneity among profiles of clinical and demographic risk indicators among patients who convert to psychosis.
To maximize clinical utility, we require an approach that can be applied to scale the risk in an individual patient at the initial clinical contact. Such individualized risk calculation is possible when a large data set on a reference population is available from which risks can be calculated based on one or more predictor variables. Well-performing risk calculators have been developed in numerous somatic disease contexts, including cardiovascular disease and cancer (5–9), where they provide a rationale for clinicians to pursue more or less invasive intervention strategies, based on the level of risk implied by an individual’s profile across a set of risk factors. They also inform patients and their family members, thus helping them make complex treatment decisions.
Here we present such an individualized risk calculator for psychosis, using data from the second phase of the North American Prodrome Longitudinal Study (NAPLS-2). Predictors were chosen a priori based on a review of the literature on psychosis risk prediction in clinical high-risk samples, blindly with respect to the empirical relationships between any of the nominated variables and psychosis outcome within the NAPLS-2 data set. We limited our scope to clinical, cognitive, and demographic measures that are readily obtainable in standard clinical settings. Using time-to-event proportional hazards regression, a risk calculator was generated that calculates risk according to an individual’s values on the included variables. We evaluated the performance of the risk calculator using the concordance index (the Harrell C-index, a measure of overall accuracy, analogous to area under the receiver operating characteristic curve) and assessed the relative importance of each of the included predictor variables.
Method
Study Subjects and Clinical Characterization
The study protocol and consent form were reviewed and approved by the institutional review boards of the eight data collection sites (UCLA, Emory, Beth Israel Deaconess Medical Center, Zucker Hillside Hospital, University of North Carolina, UCSD, Calgary, and Yale). We have previously reported on the methods for evaluation of subjects and data collection (10). Participants were evaluated using the Structured Interview for Prodromal Syndromes (SIPS) (11) and the Structured Clinical Interview for DSM-IV Axis I Disorders (12) by trained interviewers who met high reliability standards (intraclass correlation coefficients, 0.92–0.96) (10). Patients who had ever met DSM-IV criteria for a psychotic disorder, had a history of substance dependence, had a neurological disorder, or had an estimated IQ <70 were excluded. Participants met SIPS criteria (13) for the presence of one or more clinical high-risk syndromes: attenuated psychotic symptoms syndrome, brief intermittent psychotic symptom syndrome, and familial risk and deterioration syndrome.
Follow-up clinical evaluations were scheduled every 6 months after study entry through 2 years. Conversion to psychosis was determined by SIPS criteria, which are designed to operationalize the threshold of delusional ideation or hallucination severity required for a DSM-IV diagnosis of a psychotic disorder. Participants were followed up to the time of conversion to psychosis or the last contact (up to 2 years), the dates for which were recorded, permitting calculation of length of time in the study until conversion or censoring (loss to follow-up).
A total of 743 clinical high-risk patients who met SIPS criteria were enrolled in NAPLS-2 from 2008 to 2013. In the present analysis, we excluded subjects who dropped out of the study before any clinical follow-up was conducted (N=147). The final study cohort consisted of 596 clinical high-risk participants who had at least one follow-up evaluation.
Selection of Predictor Variables
To meet the objective of developing a practical tool for risk prediction, our focus was on demographic, clinical, neurocognitive, and functioning measures that are easily administered in general clinical settings. The number of predictors was limited a priori to eight to ensure that there were a minimum of 10 converters per predictor in the model, which helps mitigate model instability due to overfitting. For the same reason, we avoided including terms for the interactions among the predictors. The NAPLS-2 data set itself was not used to select predictors; doing so would have invalidated the logic of using the underlying data to inform prediction for new cases (i.e., the predictive logic would then be circular). Rather, we evaluated the published literature on psychosis prediction in clinical high-risk samples. Our selection of indicators was based on empirical links to psychosis prediction in two or more prior studies of clinical high-risk cases; there was no attempt to select predictors based on a theoretical model of causes of psychosis or clinical knowledge or intuition. Based on this process, eight variables were chosen for inclusion.
Age at ascertainment was included to help account for variation in age at onset of psychosis (14) and in processes that undergo developmental modification during the age range of our sample (15–17). Greater severity of SIPS items P1 and P2 (unusual thought content and suspiciousness) are strongly predictive of psychosis in clinical high-risk samples (3, 4). Given that the meanings of gradations below the prodromal threshold are likely to be different from those at or above this threshold, these items were modified such that all levels in the nonprodromal range (0–2 on the original scale) were redefined as 0, levels in the prodromal range (3–5 on the original scale) were redefined as 1–3, and psychotic intensity (6 on the original scale) was redefined as 4, and these scores were summed. Several studies have found that slower processing speed and lower verbal learning and memory functioning are predictive of psychosis (18, 19) and in meta-analyses have among the largest effect sizes among converters to psychosis (20). These constructs were represented by scores on the Brief Assessment of Cognition in Schizophrenia symbol coding test (21) and the Hopkins Verbal Learning Test–Revised (sum of trials 1–3) (22), respectively. Many clinical high-risk cases who convert to psychosis show a pronounced decline in social functioning in the year prior to ascertainment (23), measured here using the Global Functioning: Social scale (24). Stressful life events, along with childhood traumas, have been shown to be predictive of psychosis in studies of clinical high-risk samples (25). To represent the former, we aggregated 31 life events designated as negative and potentially relevant to individuals 12–35 years old from the Research Interview Life Events Scale (26), and for the latter, we used the Childhood Trauma and Abuse Scale (27). Family history of psychotic disorder in a first-degree relative is by itself not a robust predictor of psychosis in studies of clinical high-risk samples (3, 4), but it was nevertheless included because it elevates risk by almost 10-fold compared with the general population (28).
Statistical Methods
Risk calculators have been developed to assist health care professionals for a variety of illnesses (5–9) (see http://www.lerner.ccf.org/qhs/risk_calculator/ for examples). Calculators can derive a risk prediction for a particular person from a given set of indicators by querying a multivariate model based on a large sample of similar cases. Through imputation, calculators can accommodate incomplete information on the panel of risk indicators. However, the more complete the information available on a given case, the more powerful they become, with a tighter range of certainty.
We built a multivariate proportional hazards model to predict the likelihood of conversion to psychosis based on each participant’s demographic, cognitive, and clinical characteristics, as defined above. We tested restricted cubic splines in relation to continuous variables; as none were significant, no adjustments were made. As shown in Table 1, there were little or no missing data for age, symptom severity, family history, and social functioning. Cognitive test data were missing in less than 4% of cases, and data regarding stressful life events or traumas were missing in 12%−14% of cases. In order to reduce selection bias and maximize the sample size, missing predictors were multiply imputed with the multivariate imputation by chained equations method before the multivariate regression.
| Variable | Followed (N=596) | Not Followed (N=147) | Statistical Analysis | Missing Valuesb | ||||
|---|---|---|---|---|---|---|---|---|
| Mean | SD | Mean | SD | t | p | N | % | |
| Age (years) | 18.5 | 4.3 | 18.8 | 4.2 | –0.88 | 0.38 | 0 | 0.0 |
| Modifiedc SIPS items P1 + P2, summed score | 2.6 | 1.6 | 2.6 | 1.6 | 0.07 | 0.94 | 0 | 0.0 |
| BACS symbol coding, raw score (number completed) | 56.8 | 13.1 | 57.9 | 11.6 | –0.81 | 0.42 | 22 | 3.7 |
| Hopkins Verbal Learning Test–Revised, trials 1–3 summed | 25.6 | 5.2 | 25.1 | 5.4 | 0.85 | 0.39 | 21 | 3.5 |
| Stressful life events | 10.5 | 5.5 | 10.0 | 5.6 | 0.90 | 0.37 | 69 | 11.6 |
| N | % | N | % | χ2 | p | N | % | |
| Family history of psychosis | 96 | 16.1 | 18 | 12.2 | 1.38 | 0.24 | 2 | 0.3 |
| Decline in functioning >0 on the Global Functioning: Social scale | 270 | 45.4 | 75 | 53.5 | 3.05 | 0.08 | 1 | 0.2 |
| Traumas >1 | 289 | 56.2 | 48 | 48.5 | 2.00 | 0.16 | 82 | 13.7 |
| Male | 344 | 57.7 | 77 | 52.4 | 1.36 | 0.24 | 0 | 0.0 |
a
BACS=Brief Assessment of Cognition in Schizophrenia; NAPLS-2=second phase of the North American Prodrome Longitudinal Study; SIPS=Structured Interview for Prodromal Syndromes.
b
Among those followed. Missing values were multiply imputed with the multivariate imputation by chained equations method prior to use in prediction analyses.
c
Modified such that all levels in the nonprodromal range (0–2 on the original scale) are recoded as 0, levels in the prodromal range (3–5 on the original scale) are recoded as 1–3, and psychotic intensity (6 on the original scale) is recoded as 4.
The statistical model was internally validated using 1,000 bootstrap resamples, where the discrimination and calibration performance were evaluated. Harrell’s C-index was used to quantify the discrimination ability for separating psychosis converters and nonconverters, which is analogous to the area under the receiver operating characteristic curve, with a range of 0.5 (no discrimination) to 1 (perfect discrimination), but tailored for censored data (29). A plot of the model-predicted probabilities versus the observed outcomes was used to assess calibration performance.
All statistical analyses were conducted using R, version 3.0.1 (R Core Team, 2013) including the rms and Hmisc packages.
Results
Baseline patient characteristics are summarized in Table 1. There were no significant differences between those who were and were not followed up clinically on any of the predictor variables. Of the 596 participants for whom follow-up data were available, 84 converted to psychosis within the 2-year study period. The mean age of the sample was 18.5 years. Among converters, the mean time from baseline to conversion was 7.3 months, and among nonconverters, the mean follow-up time from baseline to the last contact was 19.1 months. A total of 280 cases were followed up at 24 months without converting, and the remaining “nonconverters” were lost to follow-up at various points between 6 months and 24 months.
The 2-year probability of conversion to psychosis was 0.16 (95% CI=0.13, 0.19). Figure 1 provides frequency distributions of predicted risks for converters and nonconverters. Converters occur at a proportionally higher rate than nonconverters in each successive risk class, beginning at a predicted risk of 0.20. The output of the multivariate proportional hazards model is presented in Table 2. Prodromal symptom severity (SIPS items P1 and P2, modified and summed), decline in social functioning, and verbal learning and memory (Hopkins Verbal Learning Test–Revised scores) were significant predictors, with nonsignificant effects for age at baseline and speed of processing (symbol coding score) (p values, <0.10), although all of these variables were significant in univariate analyses (p values, <0.01). Stressful life events, traumas, and family history of schizophrenia were not significant predictors in univariate or multivariate analyses.
FIGURE 1. Frequency Distributions of Predicted Risks Among Nonconverters and Convertersa
a Beginning at a predicted risk of 0.20 or higher, there are proportionally more converters than nonconverters in each successive risk class.
| Multivariate Model | C-Indexb | ||||
|---|---|---|---|---|---|
| Predictor | Hazard Ratio | 95% CI | p | Decrement If Removed | Increase If Added |
| Modified SIPS items P1 + P2 | 2.1 | 1.6–2.7 | <0.001 | 0.092 | N/Ac |
| Decline in social functioning (Global Functioning: Social scale) | 1.3 | 1.1–1.5 | 0.01 | 0.014 | 0.015 |
| Hopkins Verbal Learning Test–Revised, trials 1–3 summed | 0.8 | 0.6–0.9 | 0.05 | 0.007 | 0.029 |
| BACS symbol coding, raw score (number completed) | 0.8 | 0.5–1.1 | 0.10 | 0.006 | 0.033 |
| Age | 0.7 | 0.5–1.1 | 0.09 | 0.004 | 0.012 |
| Stressful life events | 1.2 | 0.9–1.6 | 0.21 | 0.001 | –0.004 |
| Family history of psychosis | 1.2 | 0.7–2.1 | 0.55 | 0.000 | 0.001 |
| Traumas | 1.0 | 0.8–1.3 | 0.99 | –0.004 | 0.002 |
a
BACS=Brief Assessment of Cognition in Schizophrenia; SIPS=Structured Interview for Prodromal Syndromes.
b
Harrell’s C-index (equivalent to the area under the receiver operating characteristic curve) was used to quantify the discrimination ability for separating converters and nonconverters. The C-index for the overall model was 0.714.
c
The base model included only the modified SIPS P1 + P2 score; the C-index for the base model was 0.666.
Table 2 provides additional diagnostics of the performance of individual predictor variables. Predictors associated with the largest decreases in the C-index when removed from the model were symptom severity, decline in global social functioning, Hopkins Verbal Learning Test–Revised score, and symbol coding score. Predictors associated with the largest increases in the C-index (i.e., above that of the base model, which included only symptom severity) were symbol coding score, Hopkins Verbal Learning Test–Revised score, decline in social functioning, and age. Family history of psychosis, stressful life events, and traumas did not alter the C-index by more than 0.5% when added to or deleted from the model.
Based on the bootstrap internal validation, the multivariate model achieved a C-index of 0.71. As shown in Figure 2, the calibration plot revealed a high degree of consistency between observed probabilities and model-predicted probabilities of conversion to psychosis within the range of 0.0–0.4, within which 95% of the cases fell (mean=0.18, SD=0.11, median=0.16). Table 3 provides statistics for prediction of actual conversion to psychosis across several thresholds of model-predicted risk. There is a trade-off between the positive predictive value (proportion of cases at the threshold of predicted risk who actually converted) and sensitivity (proportion of actual converters who had predicted risks at that threshold). The positive predictive value is maximal (48.4%) at a threshold of 0.4 of model-predicted risk, but only 17.9% of converters had model-predicted risks at this threshold. Conversely, at a model-predicted risk of 0.2 or higher, the positive predictive value is 28.1%, but with a sensitivity of 66.7%.
FIGURE 2. Calibration Plot of the Accuracy of Model-Predicted Probability in Relation to Observed Probability of Psychosisa
a The observed probability was estimated using proportional hazards regression evaluating the predicted 2-year probabilities in relation to the observed conversion events, taking into account time to conversion or censoring. The overfitting bias for the estimated observed probability was corrected using 1,000 bootstrap resamples. The plot shows excellent calibration across predicted probabilities of 0.0–0.4, corresponding to 95% of the NAPLS-2 sample. Predicted probabilities above 0.4 are too sparsely represented to permit adequate calibration testing.
| Individual’s Predicted Risk | Base Rate of Predicted Risk Class | Positive Predictive Value | Negative Predictive Value | Sensitivity | Specificity |
|---|---|---|---|---|---|
| 0.05–1.00 | 97.5 | 14.3 | 93.3 | 98.8 | 2.7 |
| 0.10–1.00 | 78.9 | 16.8 | 96.0 | 94.1 | 23.6 |
| 0.15–1.00 | 52.9 | 21.6 | 94.3 | 81.0 | 51.8 |
| 0.20–1.00 | 33.4 | 28.1 | 93.0 | 66.7 | 72.1 |
| 0.25–1.00 | 20.6 | 32.5 | 90.7 | 47.6 | 83.8 |
| 0.30–1.00 | 12.4 | 39.2 | 89.5 | 34.5 | 91.2 |
| 0.35–1.00 | 8.1 | 41.7 | 88.3 | 23.8 | 94.5 |
| 0.40–1.00 | 5.2 | 48.4 | 87.8 | 17.9 | 96.9 |
| 0.45–1.00 | 3.5 | 47.6 | 87.1 | 11.9 | 97.9 |
| 0.50–1.00 | 2.0 | 41.7 | 86.5 | 6.0 | 98.6 |
| 0.55–1.00 | 1.2 | 28.6 | 86.1 | 2.4 | 99.0 |
| 0.60–1.00 | 1.0 | 16.7 | 85.9 | 1.2 | 99.0 |
An online version of the risk calculator was built to facilitate numeric calculation of the predicted probability of conversion to psychosis (http://riskcalc.org:3838/napls/).
Discussion
The goal of this study was to develop a practical tool for the individualized prediction of psychosis in clinical high-risk patients. A well-performing risk calculator was generated from the NAPLS-2 cohort data using a small number of demographic (age, family history of psychosis), clinical (unusual thought content and suspiciousness), neurocognitive (verbal learning and memory, speed of processing), and psychosocial (traumas, stressful life events, decline in social functioning) predictor variables. The overall model achieved a C-index of 0.71, which is in the range of values for established calculators currently in use for cardiovascular disease and cancer recurrence risk, which range from 0.58 to 0.81 (5–9).
The risk calculator generates a number representing the probability of transition to psychosis given a particular profile of input variables. Technically, this is an observed likelihood of conversion within the NAPLS-2 cohort itself, but this framework uses the logic of predictive inference to extend that observed likelihood based on past cases to the predicted probability for a newly ascertained case with the same profile. This logic rests on the assumption that the new case is ascertained from the same population and in a manner similar to those in NAPLS-2.
In particular, given that this risk calculator assumes a SIPS-based diagnosis of a prodromal risk syndrome as a starting point, the risk prediction tool would not be usable if such a risk syndrome has not been diagnosed. The risk calculator also assumes particular pathways to ascertainment, in that clinical high-risk cases in NAPLS are distressed and treatment seeking. This tool would thus be most useful to clinicians with training in psychosis risk detection using the SIPS (which, in addition to risk status, ascertains severity of unusual thought content and suspiciousness and family history of psychosis), who could then use the calculator for patients who have screened positive for a prodromal risk syndrome. Critically, risk determinations should be communicated to clients by trained clinicians who can help clients understand the meaning of the risk estimates (i.e., calibrated to the sample from which they were generated) and provide commensurate treatment recommendations. Note that within the context of NAPLS-2, with a mean predicted risk of 0.18 (SD=0.11), predicted risks of 0.3 or higher are relatively rare (12.4% prevalence among those meeting clinical high-risk criteria) and potent (39.2% positive predictive value). Proper training in the administration and scoring of the other measures included in the risk calculator (symbol coding, Hopkins Verbal Learning Test–Revised, Global Functioning: Social scale, Research Interview Life Events Scale, Childhood Trauma and Abuse Scale) is also required.
A key advantage of the risk calculator is that it inherently accommodates heterogeneity in profiles of risk factors among clinical high-risk cases. Examining configurations that vary across the significant predictors—greater prodromal symptom severity, lower verbal learning and memory, slower speed of processing, greater decline in social functioning, and younger age—reveals that dozens of separate permutations yield predicted conversion risks of 0.3 or higher. Stressful life events, traumas, and family history of schizophrenia have a negligible impact on their own or in combinations with other variables in the prediction of psychosis, but they were present more frequently among clinical high-risk individuals compared with healthy comparison subjects. Perhaps these variables are more significant for determining presence of a clinical high-risk syndrome and thus are not as sensitive to outcomes within a group of subjects with a clinical high-risk syndrome.
A crucial test of robustness of a statistical model is validation on an independent external data set. In a companion article (30), Carrión et al. report on a replication test of the NAPLS-2 risk calculator in an independent sample from the Early Detection, Intervention, and Prevention of Psychosis Program (EDIPPP) that included 176 clinical high-risk cases diagnosed using the SIPS and followed clinically to monitor conversion. Only the stress and trauma variables—found to be negligible in predicting conversion here—were not collected and were therefore omitted from the replication testing. The remaining six NAPLS-2 risk factors yielded a significant time-to-event proportional hazards regression model predicting conversion in the EDIPPP sample (p<0.003), with a C-index of 0.79, which is even better than in the NAPLS-2 sample (0.71). The predictive model was well calibrated, and the NAPLS-2 calculator provided a reasonable estimation of psychosis risk when considering the risk prediction generated by the validation model and the actual observed outcomes. In addition, when applied to the external EDIPPP sample, the NAPLS-2 calculator showed sensitivity and specificity values comparable to those observed in the NAPLS-2 sample across different levels of model-predicted risk (30).
Our findings also show some degree of convergence with previous studies that reported multivariate models but that used their own samples for variable selection (i.e., model optimization) and did not present a web-based tool for extending individualized risk estimation to future patients (3, 23, 31). For example, a recent study (23) using a smaller (N=92) and nonoverlapping sample of clinical high-risk cases from one of the NAPLS sites developed a classifier that included three of the predictors included in the NAPLS-2 risk calculator (suspiciousness, verbal memory deficits, and decline in social functioning). Note that the sample in that study was less than one-fifth the size of the present sample, used a more restricted age range (ages 12–20), and was not ascertained using SIPS criteria, factors that make risk classifications based on it much less generalizable than that of the NAPLS-2 risk calculator.
The most immediate uses of the risk calculator are likely to be in the selection of subjects for participation in clinical (prevention) trials, given the desire to avoid exposing patients with lower conversion risks to the potential adverse consequences of any interventions and given the potential to evaluate whether interventions differ in effectiveness based on initial risk levels or profiles across predictors. In terms of clinical practice outside the context of a prevention trial, at this point the most likely use of the risk calculator is for the clinician to be able to communicate to the patient and family a scaling of risk that could help in recruiting their cooperation with a monitoring and/or intervention plan.
A current limitation of the psychosis risk calculator is that risk estimates are not bounded by a confidence interval, making it unclear how well the single value output as a conversion risk represents the individual’s actual likelihood of conversion. This issue is particularly problematic for computed risks of 0.5 or higher, for which there is sparse representation in the NAPLS-2 data set and for which calibration of the risk calculator could consequently not be adequately tested. Nevertheless, the use of confidence intervals is not likely to be of value in discussing risks with individual patients and their families, as the general concept of a confidence interval relates to likelihoods under future sampling rather than to an individual case, and the calculated risk is the best estimate for that individual (32).
Because the replication study (EDIPPP) included several community behavioral health centers and intergovernmental managed mental health organizations, the risk calculator appears to be generalizable beyond academic medical centers, at least within the U.S. health care system. The degree to which the risk calculator generalizes to other health care system models remains an open question.
In addition to testing the calculator’s performance in independent data sets, future work could determine whether other variables, including biological tests, can improve prediction over and above the set of clinical, demographic, and cognitive measures evaluated here. Some promising leads on the use of biological assays to predict psychosis among clinical high-risk patients have emerged using empirically based discovery approaches, including machine learning algorithms for gray matter variations in structural brain images (33) and “greedy” regression algorithms for proteomic/metabolic plasma parameters (34). Studies employing discovery-oriented model-optimization methods, with parallel, independent samples, are needed to better inform future versions of this and other risk calculators. However, it is still critical to note that the data used in any risk calculator cannot be the same data that are used in the model optimization phase; as noted above, doing so would invalidate the risk predictions for new cases.
Given that in approximately one-third of clinical high-risk cases, the symptoms that determined their initial risk status remit within 6–12 months of ascertainment (35, 36), it should be possible to develop a complementary tool to predict a new case’s likelihood of remission from a clinical high-risk syndrome. Such an estimate would not necessarily be merely the inverse of the conversion risk, as different predictors may be relevant.
It is also possible that risk calculators could eventually be used to select clients for different treatment regimens or to reclassify risk after completion of a particular intervention. At this stage, the knowledge base for doing so is limited, as only a small number of controlled prevention trials in clinical high-risk cases have appeared. Collectively, the results support the view that any targeted intervention, whether biological or psychological in approach, is associated with better outcomes than less targeted control conditions (37). Results of two small trials with antipsychotic drugs do not support a prophylactic effect on conversion risk beyond the period of active treatment (38, 39). In general, the use of such medications in individuals who are below the threshold of full psychosis is not recommended. Intriguing results have been obtained in an initial trial of omega-3 fatty acid supplementation (40); this finding awaits confirmation by independent studies. Psychosocial interventions such as cognitive-behavior therapy and family-focused psychoeducation may be beneficial in deflecting the course of illness severity and chronicity (41, 42); however, it remains unclear whether such approaches can prevent onset of illness. Future intervention studies are encouraged to use the risk calculator at end-stage analysis to determine whether treatment efficacy is moderated by initial risk level or profile.
Ultimately, the degree of risk estimated by the risk calculator may be useful for weighing the cost-benefit ratios of various treatment options that emerge from clinical intervention research in the clinical high-risk population. Treatments associated with greater risks to the patient (e.g., medication side effects) or greater costs to health care delivery systems (e.g., resource- and time-intensive psychotherapeutic inventions) may best be reserved for those with higher-than-median levels of predicted risk (i.e., ≥0.16), while cost-effective treatments with benign side effect profiles may be the best option for those whose predicted risk for psychosis is in the lower range.
Like a person at risk for cardiovascular disease or cancer, an individual with a prodromal risk syndrome is more interested in receiving information pertinent to his or her personal risk profile than information about the population at large. Publication of this risk calculator is intended to assist clinicians in providing such personalized risk estimates. It is of course possible for untrained individuals to access these tools and approximate their scores on the set of risk variables. If, in so doing, a high predicted risk of conversion were generated, this could lead to significant personal distress. To mitigate this possibility, we have built in a decision tree for the online calculator that requires confirmation of an interview-based SIPS diagnosis of a prodromal risk syndrome and confirmation that the ratings and test scores were obtained by a professional; if either one of these verifications are missing, the decision tree opts out of making a prediction. The risk for loss of privacy or stigmatization based on access of the prediction tool by untrained users is also mitigated for these reasons.
In summary, a well-performing risk calculator for psychosis is available for application to new patients who meet criteria for a psychosis risk syndrome. Challenges to be addressed in the next phase of research include incorporating biological assays into the risk calculations, extending the analysis to predict likelihood of remission, extending the framework to calculate reductions in risk based on particular interventions, and investigating how patients and family members feel about and use this information.
Footnote
Dr. Cannon has served as a consultant for the Los Angeles County Department of Mental Health and Boehringer-Ingelheim Pharmaceuticals. Dr. Mathalon has served as a consultant for Boehringer-Ingelheim Pharmaceuticals. Dr. Perkins has served as a consultant for Genentech, Lundbeck, Otsuka, and Sunovion, has participated in educational activity for Otsuka, and has received research support from Genentech. Dr. Woods has received investigator-initiated research support from Pfizer and sponsor-initiated research support from Auspex and Teva; he has served as a consultant for Biomedisyn (unpaid), Boehringer-Ingelheim, and Merck and as an unpaid consultant to DSM-5; he has been granted a patent for a method of treating prodromal schizophrenia with glycine agonists and is named as an inventor on a patent pending for a method of predicting psychosis risk using blood biomarker analysis; and he has received royalties from Oxford University Press. The other authors report no financial relationships with commercial interests.
References
1.
Insel TR: The arrival of preemptive psychiatry. Early Interv Psychiatry 2007; 1:5–6
2.
Fusar-Poli P, Bonoldi I, Yung AR, et al: Predicting psychosis: meta-analysis of transition outcomes in individuals at high clinical risk. Arch Gen Psychiatry 2012; 69:220–229
3.
Cannon TD, Cadenhead K, Cornblatt B, et al: Prediction of psychosis in youth at high clinical risk: a multisite longitudinal study in North America. Arch Gen Psychiatry 2008; 65:28–37
4.
Fusar-Poli P, Borgwardt S, Bechdolf A, et al: The psychosis high-risk state: a comprehensive state-of-the-art review. JAMA Psychiatry 2013; 70:107–120
5.
Kattan MW, Yu C, Stephenson AJ, et al: Clinicians versus nomogram: predicting future technetium-99m bone scan positivity in patients with rising prostate-specific antigen after radical prostatectomy for prostate cancer. Urology 2013; 81:956–961
6.
Lee TH, Marcantonio ER, Mangione CM, et al: Derivation and prospective validation of a simple index for prediction of cardiac risk of major noncardiac surgery. Circulation 1999; 100:1043–1049
7.
Pfeiffer RM, Park Y, Kreimer AR, et al: Risk prediction for breast, endometrial, and ovarian cancer in white women aged 50 y or older: derivation and validation from population-based cohort studies. PLoS Med 2013; 10:e1001492
8.
Ross PL, Gerigk C, Gonen M, et al: Comparisons of nomograms and urologists’ predictions in prostate cancer. Semin Urol Oncol 2002; 20:82–88
9.
Specht MC, Kattan MW, Gonen M, et al: Predicting nonsentinel node status after positive sentinel lymph biopsy for breast cancer: clinicians versus nomogram. Ann Surg Oncol 2005; 12:654–659
10.
Addington J, Cadenhead KS, Cornblatt BA, et al: North American Prodrome Longitudinal Study (NAPLS 2): overview and recruitment. Schizophr Res 2012; 142:77–82
11.
McGlashan TH, Walsh BC, Woods SW: The Psychosis-Risk Syndrome: Handbook for Diagnosis and Follow-Up. New York, Oxford University Press, 2010
12.
First M, Spitzer RL, Gibbon M, et al: Structured Clinical Interview for DSM-IV Axis I Disorders, Patient Edition. New York, Biometrics Research Department, New York State Psychiatric Institute, 1995
13.
Miller TJ, McGlashan TH, Rosen JL, et al: Prospective diagnosis of the initial prodrome for schizophrenia based on the Structured Interview for Prodromal Syndromes: preliminary evidence of interrater reliability and predictive validity. Am J Psychiatry 2002; 159:863–865
14.
Kirkbride JB, Fearon P, Morgan C, et al: Heterogeneity in incidence rates of schizophrenia and other psychotic syndromes: findings from the 3-center AeSOP study. Arch Gen Psychiatry 2006; 63:250–258
15.
Cannon TD, Chung Y, He G, et al: Progressive reduction in cortical thickness as psychosis develops: a multisite longitudinal neuroimaging study of youth at elevated clinical risk. Biol Psychiatry 2015; 77:147–157
16.
McGlashan TH, Hoffman RE: Schizophrenia as a disorder of developmentally reduced synaptic connectivity. Arch Gen Psychiatry 2000; 57:637–648
17.
Walker EF, Walder DJ, Reynolds F: Developmental changes in cortisol secretion in normal and at-risk youth. Dev Psychopathol 2001; 13:721–732
18.
Riecher-Rössler A, Pflueger MO, Aston J, et al: Efficacy of using cognitive status in predicting psychosis: a 7-year follow-up. Biol Psychiatry 2009; 66:1023–1030
19.
Seidman LJ, Giuliano AJ, Meyer EC, et al: Neuropsychology of the prodrome to psychosis in the NAPLS consortium: relationship to family history and conversion to psychosis. Arch Gen Psychiatry 2010; 67:578–588
20.
Giuliano AJ, Li H, Mesholam-Gately RI, et al: Neurocognition in the psychosis risk syndrome: a quantitative and qualitative review. Curr Pharm Des 2012; 18:399–415
21.
Keefe RS, Goldberg TE, Harvey PD, et al: The Brief Assessment of Cognition in Schizophrenia: reliability, sensitivity, and comparison with a standard neurocognitive battery. Schizophr Res 2004; 68:283–297
22.
Brandt J, Benedict RHB: Hopkins Verbal Learning Test–Revised (HVLT-R). Odessa, Fla, Psychological Assessment Resources, 1998
23.
Cornblatt BA, Carrion RE, Auther A, et al: Psychosis prevention: a modified clinical high risk perspective from the Recognition and Prevention (RAP) Program. Am J Psychiatry 2015; 172:986–994
24.
Cornblatt BA, Auther AM, Niendam T, et al: Preliminary findings for two new measures of social and role functioning in the prodromal phase of schizophrenia. Schizophr Bull 2007; 33:688–702
25.
Holtzman CW, Shapiro DI, Trotman HD, et al: Stress and the prodromal phase of psychosis. Curr Pharm Des 2012; 18:527–533
26.
Dohrenwend BS, Krasnoff L, Askenasy AR, et al: Exemplification of a method for scaling life events: the Peri Life Events Scale. J Health Soc Behav 1978; 19:205–229
27.
Janssen I, Krabbendam L, Bak M, et al: Childhood abuse as a risk factor for psychotic experiences. Acta Psychiatr Scand 2004; 109:38–45
28.
Sullivan PF: The genetics of schizophrenia. PLoS Med 2005; 2:e212
29.
Uno H, Cai T, Pencina MJ, et al: On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Stat Med 2011; 30:1105–1117
30.
Carrión RE, Cornblatt BA, Burton CZ, et al: Personalized prediction of psychosis: external validation of the NAPLS-2 psychosis risk calculator with the EDIPPP project. Am J Psychiatry 2016; 173:989–996
31.
Ruhrmann S, Schultze-Lutter F, Salokangas RK, et al: Prediction of psychosis in adolescents and young adults at high risk: results from the prospective European Prediction of Psychosis Study. Arch Gen Psychiatry 2010; 67:241–251
32.
Kattan MW: Doc, what are my chances? A conversation about prognostic uncertainty. Eur Urol 2011; 59:224
33.
Koutsouleris N, Riecher-Rossler A, Meisenzahl EM, et al: Detecting the psychosis prodrome across high-risk populations using neuroanatomical biomarkers. Schizophr Bull 2015; 41:471–482
34.
Perkins DO, Jeffries CD, Addington J, et al: Towards a psychosis risk blood diagnostic for persons experiencing high-risk symptoms: preliminary results from the NAPLS project. Schizophr Bull 2015; 41:419–428
35.
Addington J, Cornblatt BA, Cadenhead KS, et al: At clinical high risk for psychosis: outcome for nonconverters. Am J Psychiatry 2011; 168:800–805
36.
Schlosser DA, Jacobson S, Chen Q, et al: Recovery from an at-risk state: clinical and functional outcomes of putatively prodromal youth who do not develop psychosis. Schizophr Bull 2012; 38:1225–1233
37.
van der Gaag M, Smit F, Bechdolf A, et al: Preventing a first episode of psychosis: meta-analysis of randomized controlled prevention trials of 12 month and longer-term follow-ups. Schizophr Res 2013; 149:56–62
38.
McGorry PD, Yung AR, Phillips LJ, et al: Randomized controlled trial of interventions designed to reduce the risk of progression to first-episode psychosis in a clinical sample with subthreshold symptoms. Arch Gen Psychiatry 2002; 59:921–928
39.
McGlashan TH, Zipursky RB, Perkins D, et al: Randomized, double-blind trial of olanzapine versus placebo in patients prodromally symptomatic for psychosis. Am J Psychiatry 2006; 163:790–799
40.
Amminger GP, Schäfer MR, Papageorgiou K, et al: Long-chain omega-3 fatty acids for indicated prevention of psychotic disorders: a randomized, placebo-controlled trial. Arch Gen Psychiatry 2010; 67:146–154
41.
Miklowitz DJ, O’Brien MP, Schlosser DA, et al: Family-focused treatment for adolescents and young adults at high risk for psychosis: results of a randomized trial. J Am Acad Child Adolesc Psychiatry 2014; 53:848–858
42.
Stafford MR, Jackson H, Mayo-Wilson E, et al: Early interventions to prevent psychosis: systematic review and meta-analysis. BMJ 2013; 346:f185
Information & Authors
Information
Published In
History
Received: 8 July 2015
Revision received: 14 December 2015
Revision received: 8 March 2016
Accepted: 30 March 2016
Published online: 1 July 2016
Published in print: October 01, 2016
Authors
Funding Information
National Institute of Mental Health10.13039/100000025: P50MH066286, U01MH081857, U01MH081902
Supported by NIH grants U01 MH081902 (to Dr. Cannon), P50 MH066286 (to Dr. Bearden), U01 MH081857 (to Dr. Cornblatt), U01 MH82022 (to Dr. Woods), U01 MH066134 (to Dr. Addington), U01 MH081944 (to Dr. Cadenhead), R01 U01 MH066069 (to Dr. Perkins), R01 MH076989 (to Dr. Mathalon), U01 MH081928 (to Dr. Seidman), and U01 MH081988 (to Dr. Walker).Metrics & Citations
Metrics
Citations
Export Citations
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
For more information or tips please see 'Downloading to a citation manager' in the Help menu.
View Options
View options
PDF/EPUB
View PDF/EPUBLogin options
Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.
Personal login Institutional Login Open Athens loginNot a subscriber?
PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.
Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).