Validating the Predictive Accuracy of the NAPLS-2 Psychosis Risk Calculator in a Clinical High-Risk Sample From the SHARP (Shanghai At Risk for Psychosis) Program

To the Editor: A web-based risk calculator (http://riskcalc.org:3838/napls/) for use in clinical high-risk populations was developed in the second phase of the North American Prodrome Longitudinal Study (NAPLS-2) (1). This calculator integrated baseline age, unusual thoughts and suspiciousness, symbol coding, verbal learning test performance, functional decline, and family history of psychosis variables and achieved a concordance index of 0.71 for predicting psychosis. A study in an independent U.S. sample validated the risk calculator and provided supporting evidence for its application and dissemination (2). Should robust cross-validations occur in different countries with different populations, this would strengthen its potential use clinically in early identification and intervention programs treating individual clinical high-risk cases. There are many steps needed before such tools can be implemented. At this point, cross-validation in other independent samples is an important step that would strengthen the evidence base for use of the risk calculator.

An important question is how the NAPLS-2 risk calculator will work in other parts of the world, such as in Asian samples, that have different cultural and social backgrounds. From a validity standpoint, it is ideal for such replications to measure the same risk factors using very similar inclusion criteria and assessments comparable to the NAPLS. Such a study does exist. In 2010, the Shanghai At Risk for Psychosis (SHARP) study was launched at the Shanghai Mental Health Center, the largest outpatient mental health clinic in China (3, 4). The Chinese SHARP research and clinical team has been working closely with a U.S. team that was led by Larry J. Seidman, Ph.D., who was also the principal investigator at the Harvard Medical School site of the NAPLS project. Together, these teams have implemented methods similar to those used in the NAPLS for the identification of clinical high-risk individuals in mainland China in studies jointly funded by the National Institute of Mental Health and Chinese agencies.

A total of 300 clinical high-risk youths were identified using the Structured Interview for Prodromal Syndromes (SIPS). Among them, 228 (76.0%) completed neurocognitive assessments at baseline, 199 (87.3%) clinical high-risk youths had at least a 1-year follow-up assessment, and 46 (23.1%) converted to full psychosis. Details of the study procedures, study setting, implementation of the measurement, and assessment are reported elsewhere (3, 4). The clinical high-risk youths in the SHARP and NAPLS-2 samples were compared on demographic and clinical variables (Table 1). The six key predictor variables were entered into the NAPLS-2 risk calculator by two persons independently, and a new risk ratio variable for the Chinese clinical high-risk population was constructed. The only difference was that the Global Functioning: Social scale in the NAPLS-2 risk calculator was replaced by the Global Assessment of Functioning Scale (GAF) change score, which also measures functional deterioration (score relative to the previous 12 months). A GAF score that has declined to 5% or less of the previous best GAF score is recoded as 0. Declines of 5%–15% are recoded as 1, 15%−25% as 2, 25%−35% as 3, 35%−45% as 4, 45%−55% as 5, and 55%−65% as 6. Our data highlight the importance of a declining GAF score in the prediction of psychosis (4); that is, we found a significant positive association (r_s=0.884, N=200, p<0.001, Spearman rank-order correlation) and comparability for predicting psychosis by the receiver operating characteristic analysis between the GAF and the Global Functioning: Social scale in later samples that were acquired. Another reason for using the GAF score is that cultural differences have not been examined and may affect the validity of social functioning scales; otherwise, the GAF scores can be derived from the SIPS assessment and have been widely used in China for many years.

We investigated whether probability risk estimates provided by the NAPLS-2 risk calculator for each individual in the SHARP validation sample could discriminate converters from nonconverters. When conversion to psychosis is the principal endpoint, the receiver operating characteristic analysis resulted in an area under the curve (AUC) value of 0.631 (95% CI=0.542–0.721, p=0.007) for the probability risk estimates. Frequency distributions of predicted risks for converters and nonconverters in the SHARP sample are in good agreement with those obtained using the NAPLS-2 risk calculator. Converters occur at a proportionally higher rate than nonconverters at a predicted risk of 0.20 (χ²=4.450, p=0.035) (Figure 1). In addition, Table 2 summarizes the performance of probability risk estimated by the NAPLS-2 risk calculator for the SHARP sample.

The aim of this study was to cross-validate the NAPLS-2 risk calculator in a Chinese clinical high-risk sample. To the best of our knowledge, this is the first attempt to verify the NAPLS-2 risk calculator using a comparable data set from an Asian sample and only the second to do so with a non-NAPLS sample, although the NAPLS-2 risk calculator did not fit our SHARP data as well as it fit the original sample. We believe that our slightly lower AUC is to be expected given a completely independent sample, which may be subject to the issue of statistical “shrinkage” (i.e., less good fit when applying regression models to new samples). This result suggests that the NAPLS-2 risk calculator has some generalizability to an Asian country and may have usefulness in clinical applications in China. This information provides a critical first step in the implementation of the NAPLS-2 risk calculator for the clinical high-risk population in China and supports the validity of the risk calculator in novel samples. However, as emphasized by Cannon et al. (1) and Carrión et al. (2) in the October 2016 issue of the Journal, the risk calculator remains experimental. At this point, it should be used only in research settings and with clinicians who have had rigorous SIPS training (SIPS scores being at the core of the model) and not yet used in general clinical settings with individuals until its clinical utility and properties are validated more firmly.