Skip to main content
Full access
Letters to the Editor
Published Online: 1 September 2018

Validating the Predictive Accuracy of the NAPLS-2 Psychosis Risk Calculator in a Clinical High-Risk Sample From the SHARP (Shanghai At Risk for Psychosis) Program

To the Editor: A web-based risk calculator (http://riskcalc.org:3838/napls/) for use in clinical high-risk populations was developed in the second phase of the North American Prodrome Longitudinal Study (NAPLS-2) (1). This calculator integrated baseline age, unusual thoughts and suspiciousness, symbol coding, verbal learning test performance, functional decline, and family history of psychosis variables and achieved a concordance index of 0.71 for predicting psychosis. A study in an independent U.S. sample validated the risk calculator and provided supporting evidence for its application and dissemination (2). Should robust cross-validations occur in different countries with different populations, this would strengthen its potential use clinically in early identification and intervention programs treating individual clinical high-risk cases. There are many steps needed before such tools can be implemented. At this point, cross-validation in other independent samples is an important step that would strengthen the evidence base for use of the risk calculator.
An important question is how the NAPLS-2 risk calculator will work in other parts of the world, such as in Asian samples, that have different cultural and social backgrounds. From a validity standpoint, it is ideal for such replications to measure the same risk factors using very similar inclusion criteria and assessments comparable to the NAPLS. Such a study does exist. In 2010, the Shanghai At Risk for Psychosis (SHARP) study was launched at the Shanghai Mental Health Center, the largest outpatient mental health clinic in China (3, 4). The Chinese SHARP research and clinical team has been working closely with a U.S. team that was led by Larry J. Seidman, Ph.D., who was also the principal investigator at the Harvard Medical School site of the NAPLS project. Together, these teams have implemented methods similar to those used in the NAPLS for the identification of clinical high-risk individuals in mainland China in studies jointly funded by the National Institute of Mental Health and Chinese agencies.
A total of 300 clinical high-risk youths were identified using the Structured Interview for Prodromal Syndromes (SIPS). Among them, 228 (76.0%) completed neurocognitive assessments at baseline, 199 (87.3%) clinical high-risk youths had at least a 1-year follow-up assessment, and 46 (23.1%) converted to full psychosis. Details of the study procedures, study setting, implementation of the measurement, and assessment are reported elsewhere (3, 4). The clinical high-risk youths in the SHARP and NAPLS-2 samples were compared on demographic and clinical variables (Table 1). The six key predictor variables were entered into the NAPLS-2 risk calculator by two persons independently, and a new risk ratio variable for the Chinese clinical high-risk population was constructed. The only difference was that the Global Functioning: Social scale in the NAPLS-2 risk calculator was replaced by the Global Assessment of Functioning Scale (GAF) change score, which also measures functional deterioration (score relative to the previous 12 months). A GAF score that has declined to 5% or less of the previous best GAF score is recoded as 0. Declines of 5%–15% are recoded as 1, 15%−25% as 2, 25%−35% as 3, 35%−45% as 4, 45%−55% as 5, and 55%−65% as 6. Our data highlight the importance of a declining GAF score in the prediction of psychosis (4); that is, we found a significant positive association (rs=0.884, N=200, p<0.001, Spearman rank-order correlation) and comparability for predicting psychosis by the receiver operating characteristic analysis between the GAF and the Global Functioning: Social scale in later samples that were acquired. Another reason for using the GAF score is that cultural differences have not been examined and may affect the validity of social functioning scales; otherwise, the GAF scores can be derived from the SIPS assessment and have been widely used in China for many years.
TABLE 1. Comparison of Characteristics of Clinical High-Risk Subjects Who Were in the SHARP Program or in the NAPLS-2a
VariableNAPLS-2 (N=596; followed)SHARP (N=199; followed)Statistical Analysis
 MeanSDMeanSDtp
Age (years)18.54.319.15.11.6950.092
Modified P1 and P2 SIPS itemsb2.61.63.11.55.168<0.001
Hopkins Verbal Learning Test–Revised (raw score)25.65.223.55.4–5.534<0.001
Brief Assessment of Cognition in Schizophrenia symbol coding test (raw score)56.813.157.910.01.4930.137
 N%N%χ2p
Male34457.79447.26.6250.010
Family history of psychosis9616.1178.57.0010.008
a
SHARP=Shanghai At Risk for Psychosis; NAPLS-2=second phase of the North American Prodrome Longitudinal Study.
b
Represents the severity of unusual thought content and suspiciousness (items P1 and P2 in the Structured Interview for Prodromal Syndromes [SIPS]). P1 or P2 items rated 0–2 on the original scale are recoded as 0; items rated 3–6 on the original scale are recoded as 1–4.
We investigated whether probability risk estimates provided by the NAPLS-2 risk calculator for each individual in the SHARP validation sample could discriminate converters from nonconverters. When conversion to psychosis is the principal endpoint, the receiver operating characteristic analysis resulted in an area under the curve (AUC) value of 0.631 (95% CI=0.542–0.721, p=0.007) for the probability risk estimates. Frequency distributions of predicted risks for converters and nonconverters in the SHARP sample are in good agreement with those obtained using the NAPLS-2 risk calculator. Converters occur at a proportionally higher rate than nonconverters at a predicted risk of 0.20 (χ2=4.450, p=0.035) (Figure 1). In addition, Table 2 summarizes the performance of probability risk estimated by the NAPLS-2 risk calculator for the SHARP sample.
FIGURE 1. Frequency Distributions of Predicted Risk Among Nonconverters and Converters in the Shanghai At Risk for Psychosis (SHARP) Sample
TABLE 2. Psychometric Property Values of the Predicted Risk Index for Conversion to Psychosis (or Nonrecovered)
Predicted Risk (%)SensitivitySpecificityPositive Predictive ValueaNegative Predictive Valueb
≥1097.89.224.593.3
≥2071.745.828.584.3
≥3037.077.132.780.3
≥4021.786.332.378.6
≥5015.293.541.378.6
a
Positive predictive value represents the proportions of positive results in a risk class at the specified level or higher that are true positive.
b
Negative predictive value represents the proportions of negative results in a risk class at the specified level or higher that are true negative.
The aim of this study was to cross-validate the NAPLS-2 risk calculator in a Chinese clinical high-risk sample. To the best of our knowledge, this is the first attempt to verify the NAPLS-2 risk calculator using a comparable data set from an Asian sample and only the second to do so with a non-NAPLS sample, although the NAPLS-2 risk calculator did not fit our SHARP data as well as it fit the original sample. We believe that our slightly lower AUC is to be expected given a completely independent sample, which may be subject to the issue of statistical “shrinkage” (i.e., less good fit when applying regression models to new samples). This result suggests that the NAPLS-2 risk calculator has some generalizability to an Asian country and may have usefulness in clinical applications in China. This information provides a critical first step in the implementation of the NAPLS-2 risk calculator for the clinical high-risk population in China and supports the validity of the risk calculator in novel samples. However, as emphasized by Cannon et al. (1) and Carrión et al. (2) in the October 2016 issue of the Journal, the risk calculator remains experimental. At this point, it should be used only in research settings and with clinicians who have had rigorous SIPS training (SIPS scores being at the core of the model) and not yet used in general clinical settings with individuals until its clinical utility and properties are validated more firmly.

Acknowledgments

Drs. Seidman and McCarley were founders and core members of the Shanghai At Risk for Psychosis (SHARP) project.

Footnote

Dr. Seidman died in September 2017. Dr. McCarley died in May 2017.

References

1.
Cannon TD, Yu C, Addington J, et al: An individualized risk calculator for research in prodromal psychosis. Am J Psychiatry 2016; 173:980–988
2.
Carrión RE, Cornblatt BA, Burton CZ, et al: Personalized prediction of psychosis: external validation of the NAPLS-2 psychosis risk calculator with the EDIPPP project. Am J Psychiatry 2016; 173:989–996
3.
Zhang T, Li H, Woodberry KA, et al: Prodromal psychosis detection in a counseling center population in China: an epidemiological and clinical study. Schizophr Res 2014; 152:391–399
4.
Ren W, Ma J, Li J, et al: Repetitive transcranial magnetic stimulation (rTMS) modulates lipid metabolism in aging adults. Front Aging Neurosci 2017; 9:334

Information & Authors

Information

Published In

Go to American Journal of Psychiatry
Go to American Journal of Psychiatry
American Journal of Psychiatry
Pages: 906 - 908
PubMed: 30173545

History

Accepted: 18 June 2018
Published online: 1 September 2018
Published in print: September 01, 2018

Keywords

  1. Ultra High Risk
  2. Prodromal Psychosis
  3. Risk Calculator
  4. Transition
  5. Prediction

Authors

Details

TianHong Zhang, M.D., Ph.D.
From the Shanghai Mental Health Center, Shanghai Jiaotong University School of Medicine, Shanghai Key Laboratory of Psychotic Disorders, Shanghai; the Department of Psychology, Florida A&M University, Tallahassee; the Department of Psychiatry, Harvard Medical School at Beth Israel Deaconess Medical Center, Boston; the Departments of Psychiatry and Radiology, Brigham and Women’s Hospital, Boston; and the VA Boston Healthcare System, Boston.
HuiJun Li, Ph.D.
From the Shanghai Mental Health Center, Shanghai Jiaotong University School of Medicine, Shanghai Key Laboratory of Psychotic Disorders, Shanghai; the Department of Psychology, Florida A&M University, Tallahassee; the Department of Psychiatry, Harvard Medical School at Beth Israel Deaconess Medical Center, Boston; the Departments of Psychiatry and Radiology, Brigham and Women’s Hospital, Boston; and the VA Boston Healthcare System, Boston.
YingYing Tang, Ph.D.
From the Shanghai Mental Health Center, Shanghai Jiaotong University School of Medicine, Shanghai Key Laboratory of Psychotic Disorders, Shanghai; the Department of Psychology, Florida A&M University, Tallahassee; the Department of Psychiatry, Harvard Medical School at Beth Israel Deaconess Medical Center, Boston; the Departments of Psychiatry and Radiology, Brigham and Women’s Hospital, Boston; and the VA Boston Healthcare System, Boston.
Margaret A. Niznikiewicz, Ph.D.
From the Shanghai Mental Health Center, Shanghai Jiaotong University School of Medicine, Shanghai Key Laboratory of Psychotic Disorders, Shanghai; the Department of Psychology, Florida A&M University, Tallahassee; the Department of Psychiatry, Harvard Medical School at Beth Israel Deaconess Medical Center, Boston; the Departments of Psychiatry and Radiology, Brigham and Women’s Hospital, Boston; and the VA Boston Healthcare System, Boston.
Martha E. Shenton, Ph.D.
From the Shanghai Mental Health Center, Shanghai Jiaotong University School of Medicine, Shanghai Key Laboratory of Psychotic Disorders, Shanghai; the Department of Psychology, Florida A&M University, Tallahassee; the Department of Psychiatry, Harvard Medical School at Beth Israel Deaconess Medical Center, Boston; the Departments of Psychiatry and Radiology, Brigham and Women’s Hospital, Boston; and the VA Boston Healthcare System, Boston.
Matcheri S. Keshavan, M.D.
From the Shanghai Mental Health Center, Shanghai Jiaotong University School of Medicine, Shanghai Key Laboratory of Psychotic Disorders, Shanghai; the Department of Psychology, Florida A&M University, Tallahassee; the Department of Psychiatry, Harvard Medical School at Beth Israel Deaconess Medical Center, Boston; the Departments of Psychiatry and Radiology, Brigham and Women’s Hospital, Boston; and the VA Boston Healthcare System, Boston.
William S. Stone, Ph.D.
From the Shanghai Mental Health Center, Shanghai Jiaotong University School of Medicine, Shanghai Key Laboratory of Psychotic Disorders, Shanghai; the Department of Psychology, Florida A&M University, Tallahassee; the Department of Psychiatry, Harvard Medical School at Beth Israel Deaconess Medical Center, Boston; the Departments of Psychiatry and Radiology, Brigham and Women’s Hospital, Boston; and the VA Boston Healthcare System, Boston.
Robert W. McCarley, M.D.
From the Shanghai Mental Health Center, Shanghai Jiaotong University School of Medicine, Shanghai Key Laboratory of Psychotic Disorders, Shanghai; the Department of Psychology, Florida A&M University, Tallahassee; the Department of Psychiatry, Harvard Medical School at Beth Israel Deaconess Medical Center, Boston; the Departments of Psychiatry and Radiology, Brigham and Women’s Hospital, Boston; and the VA Boston Healthcare System, Boston.
Larry J. Seidman, Ph.D.
From the Shanghai Mental Health Center, Shanghai Jiaotong University School of Medicine, Shanghai Key Laboratory of Psychotic Disorders, Shanghai; the Department of Psychology, Florida A&M University, Tallahassee; the Department of Psychiatry, Harvard Medical School at Beth Israel Deaconess Medical Center, Boston; the Departments of Psychiatry and Radiology, Brigham and Women’s Hospital, Boston; and the VA Boston Healthcare System, Boston.
JiJun Wang, M.D., Ph.D. [email protected]
From the Shanghai Mental Health Center, Shanghai Jiaotong University School of Medicine, Shanghai Key Laboratory of Psychotic Disorders, Shanghai; the Department of Psychology, Florida A&M University, Tallahassee; the Department of Psychiatry, Harvard Medical School at Beth Israel Deaconess Medical Center, Boston; the Departments of Psychiatry and Radiology, Brigham and Women’s Hospital, Boston; and the VA Boston Healthcare System, Boston.

Notes

Address correspondence to Dr. Wang ([email protected]).

Competing Interests

Funding agencies and organizations had no role in the design, analysis, interpretation, or publication of this study.

Funding Information

National Institute of Mental Health10.13039/100000025: 1R01 MH 101052-01, 1R21 MH093294-01A1
National Natural Science Foundation of China10.13039/501100001809: 81671329, 81671332
National Key R&D Program of China: 2016YFC1306803
Supported by a National Key R&D Program of China grant (2016YFC1306803) to Dr. Wang, by National Natural Science Foundation of China grants (81671329, 81671332) to Dr. Zhang and Dr. Wang, by an R21 Fogarty/NIMH grant (1R21 MH093294-01A1) to Dr. Li, and by a U.S.-China Program for Biomedical Collaborative Research grant (1R01 MH 101052-01) to Dr. Seidman.The authors report no financial relationships with commercial interests.

Metrics & Citations

Metrics

Citations

Export Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

For more information or tips please see 'Downloading to a citation manager' in the Help menu.

Format
Citation style
Style
Copy to clipboard

View Options

View options

PDF/EPUB

View PDF/EPUB

Login options

Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.

Personal login Institutional Login Open Athens login
Purchase Options

Purchase this article to access the full text.

PPV Articles - American Journal of Psychiatry

PPV Articles - American Journal of Psychiatry

Not a subscriber?

Subscribe Now / Learn More

PsychiatryOnline subscription options offer access to the DSM-5-TR® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.

Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).

Media

Figures

Other

Tables

Share

Share

Share article link

Share