Validating the Predictive Accuracy of the NAPLS-2 Psychosis Risk Calculator in a Clinical High-Risk Sample From the SHARP (Shanghai At Risk for Psychosis) Program.

Abstract

Back to table of contents Previous article Next article Letters to the EditorFull AccessValidating the Predictive Accuracy of the NAPLS-2 Psychosis Risk Calculator in a Clinical High-Risk Sample From the SHARP (Shanghai At Risk for Psychosis) ProgramTianHong Zhang, M.D., Ph.D., HuiJun Li, Ph.D., YingYing Tang, Ph.D., Margaret A. Niznikiewicz, Ph.D., Martha E. Shenton, Ph.D., Matcheri S. Keshavan, M.D., William S. Stone, Ph.D., Robert W. McCarley, M.D., Larry J. Seidman, Ph.D., JiJun Wang, M.D., Ph.D.TianHong ZhangSearch for more papers by this author, M.D., Ph.D., HuiJun LiSearch for more papers by this author, Ph.D., YingYing TangSearch for more papers by this author, Ph.D., Margaret A. NiznikiewiczSearch for more papers by this author, Ph.D., Martha E. ShentonSearch for more papers by this author, Ph.D., Matcheri S. KeshavanSearch for more papers by this author, M.D., William S. StoneSearch for more papers by this author, Ph.D., Robert W. McCarleySearch for more papers by this author, M.D., Larry J. SeidmanSearch for more papers by this author, Ph.D., JiJun WangSearch for more papers by this author, M.D., Ph.D.Published Online:1 Sep 2018https://doi.org/10.1176/appi.ajp.2018.18010036AboutSectionsPDF/EPUB ToolsAdd to favoritesDownload CitationsTrack Citations ShareShare onFacebookTwitterLinked InEmail To the Editor: A web-based risk calculator (http://riskcalc.org:3838/napls/) for use in clinical high-risk populations was developed in the second phase of the North American Prodrome Longitudinal Study (NAPLS-2) (1). This calculator integrated baseline age, unusual thoughts and suspiciousness, symbol coding, verbal learning test performance, functional decline, and family history of psychosis variables and achieved a concordance index of 0.71 for predicting psychosis. A study in an independent U.S. sample validated the risk calculator and provided supporting evidence for its application and dissemination (2). Should robust cross-validations occur in different countries with different populations, this would strengthen its potential use clinically in early identification and intervention programs treating individual clinical high-risk cases. There are many steps needed before such tools can be implemented. At this point, cross-validation in other independent samples is an important step that would strengthen the evidence base for use of the risk calculator.An important question is how the NAPLS-2 risk calculator will work in other parts of the world, such as in Asian samples, that have different cultural and social backgrounds. From a validity standpoint, it is ideal for such replications to measure the same risk factors using very similar inclusion criteria and assessments comparable to the NAPLS. Such a study does exist. In 2010, the Shanghai At Risk for Psychosis (SHARP) study was launched at the Shanghai Mental Health Center, the largest outpatient mental health clinic in China (3, 4). The Chinese SHARP research and clinical team has been working closely with a U.S. team that was led by Larry J. Seidman, Ph.D., who was also the principal investigator at the Harvard Medical School site of the NAPLS project. Together, these teams have implemented methods similar to those used in the NAPLS for the identification of clinical high-risk individuals in mainland China in studies jointly funded by the National Institute of Mental Health and Chinese agencies.A total of 300 clinical high-risk youths were identified using the Structured Interview for Prodromal Syndromes (SIPS). Among them, 228 (76.0%) completed neurocognitive assessments at baseline, 199 (87.3%) clinical high-risk youths had at least a 1-year follow-up assessment, and 46 (23.1%) converted to full psychosis. Details of the study procedures, study setting, implementation of the measurement, and assessment are reported elsewhere (3, 4). The clinical high-risk youths in the SHARP and NAPLS-2 samples were compared on demographic and clinical variables (Table 1). The six key predictor variables were entered into the NAPLS-2 risk calculator by two persons independently, and a new risk ratio variable for the Chinese clinical high-risk population was constructed. The only difference was that the Global Functioning: Social scale in the NAPLS-2 risk calculator was replaced by the Global Assessment of Functioning Scale (GAF) change score, which also measures functional deterioration (score relative to the previous 12 months). A GAF score that has declined to 5% or less of the previous best GAF score is recoded as 0. Declines of 5%–15% are recoded as 1, 15%−25% as 2, 25%−35% as 3, 35%−45% as 4, 45%−55% as 5, and 55%−65% as 6. Our data highlight the importance of a declining GAF score in the prediction of psychosis (4); that is, we found a significant positive association (rs=0.884, N=200, p<0.001, Spearman rank-order correlation) and comparability for predicting psychosis by the receiver operating characteristic analysis between the GAF and the Global Functioning: Social scale in later samples that were acquired. Another reason for using the GAF score is that cultural differences have not been examined and may affect the validity of social functioning scales; otherwise, the GAF scores can be derived from the SIPS assessment and have been widely used in China for many years.TABLE 1. Comparison of Characteristics of Clinical High-Risk Subjects Who Were in the SHARP Program or in the NAPLS-2aVariableNAPLS-2 (N=596; followed)SHARP (N=199; followed)Statistical AnalysisMeanSDMeanSDtpAge (years)18.54.319.15.11.6950.092Modified P1 and P2 SIPS itemsb2.61.63.11.55.168<0.001Hopkins Verbal Learning Test–Revised (raw score)25.65.223.55.4–5.534<0.001Brief Assessment of Cognition in Schizophrenia symbol coding test (raw score)56.813.157.910.01.4930.137N%N%χ2pMale34457.79447.26.6250.010Family history of psychosis9616.1178.57.0010.008aSHARP=Shanghai At Risk for Psychosis; NAPLS-2=second phase of the North American Prodrome Longitudinal Study.bRepresents the severity of unusual thought content and suspiciousness (items P1 and P2 in the Structured Interview for Prodromal Syndromes [SIPS]). P1 or P2 items rated 0–2 on the original scale are recoded as 0; items rated 3–6 on the original scale are recoded as 1–4.TABLE 1. Comparison of Characteristics of Clinical High-Risk Subjects Who Were in the SHARP Program or in the NAPLS-2aEnlarge tableWe investigated whether probability risk estimates provided by the NAPLS-2 risk calculator for each individual in the SHARP validation sample could discriminate converters from nonconverters. When conversion to psychosis is the principal endpoint, the receiver operating characteristic analysis resulted in an area under the curve (AUC) value of 0.631 (95% CI=0.542–0.721, p=0.007) for the probability risk estimates. Frequency distributions of predicted risks for converters and nonconverters in the SHARP sample are in good agreement with those obtained using the NAPLS-2 risk calculator. Converters occur at a proportionally higher rate than nonconverters at a predicted risk of 0.20 (χ2=4.450, p=0.035) (Figure 1). In addition, Table 2 summarizes the performance of probability risk estimated by the NAPLS-2 risk calculator for the SHARP sample.FIGURE 1. Frequency Distributions of Predicted Risk Among Nonconverters and Converters in the Shanghai At Risk for Psychosis (SHARP) SampleTABLE 2. Psychometric Property Values of the Predicted Risk Index for Conversion to Psychosis (or Nonrecovered)Predicted Risk (%)SensitivitySpecificityPositive Predictive ValueaNegative Predictive Valueb≥1097.89.224.593.3≥2071.745.828.584.3≥3037.077.132.780.3≥4021.786.332.378.6≥5015.293.541.378.6aPositive predictive value represents the proportions of positive results in a risk class at the specified level or higher that are true positive.bNegative predictive value represents the proportions of negative results in a risk class at the specified level or higher that are true negative.TABLE 2. Psychometric Property Values of the Predicted Risk Index for Conversion to Psychosis (or Nonrecovered)Enlarge tableThe aim of this study was to cross-validate the NAPLS-2 risk calculator in a Chinese clinical high-risk sample. To the best of our knowledge, this is the first attempt to verify the NAPLS-2 risk calculator using a comparable data set from an Asian sample and only the second to do so with a non-NAPLS sample, although the NAPLS-2 risk calculator did not fit our SHARP data as well as it fit the original sample. We believe that our slightly lower AUC is to be expected given a completely independent sample, which may be subject to the issue of statistical “shrinkage” (i.e., less good fit when applying regression models to new samples). This result suggests that the NAPLS-2 risk calculator has some generalizability to an Asian country and may have usefulness in clinical applications in China. This information provides a critical first step in the implementation of the NAPLS-2 risk calculator for the clinical high-risk population in China and supports the validity of the risk calculator in novel samples. However, as emphasized by Cannon et al. (1) and Carrión et al. (2) in the October 2016 issue of the Journal, the risk calculator remains experimental. At this point, it should be used only in research settings and with clinicians who have had rigorous SIPS training (SIPS scores being at the core of the model) and not yet used in general clinical settings with individuals until its clinical utility and properties are validated more firmly.From the Shanghai Mental Health Center, Shanghai Jiaotong University School of Medicine, Shanghai Key Laboratory of Psychotic Disorders, Shanghai; the Department of Psychology, Florida A&M University, Tallahassee; the Department of Psychiatry, Harvard Medical School at Beth Israel Deaconess Medical Center, Boston; the Departments of Psychiatry and Radiology, Brigham and Women’s Hospital, Boston; and the VA Boston Healthcare System, Boston.Address correspondence to Dr. Wang ([email protected]).Dr. Seidman died in September 2017. Dr. McCarley died in May 2017.Supported by a National Key R&D Program of China grant (2016YFC1306803) to Dr. Wang, by National Natural Science Foundation of China grants (81671329, 81671332) to Dr. Zhang and Dr. Wang, by an R21 Fogarty/NIMH grant (1R21 MH093294-01A1) to Dr. Li, and by a U.S.-China Program for Biomedical Collaborative Research grant (1R01 MH 101052-01) to Dr. Seidman.Funding agencies and organizations had no role in the design, analysis, interpretation, or publication of this study.The authors report no financial relationships with commercial interests.Drs. Seidman and McCarley were founders and core members of the Shanghai At Risk for Psychosis (SHARP) project.References1 Cannon TD, Yu C, Addington J, et al.: An individualized risk calculator for research in prodromal psychosis. Am J Psychiatry 2016; 173:980–988Link, Google Scholar2 Carrión RE, Cornblatt BA, Burton CZ, et al.: Personalized prediction of psychosis: external validation of the NAPLS-2 psychosis risk calculator with the EDIPPP project. Am J Psychiatry 2016; 173:989–996Link, Google Scholar3 Zhang T, Li H, Woodberry KA, et al.: Prodromal psychosis detection in a counseling center population in China: an epidemiological and clinical study. Schizophr Res 2014; 152:391–399Crossref, Medline, Google Scholar4 Ren W, Ma J, Li J, et al.: Repetitive transcranial magnetic stimulation (rTMS) modulates lipid metabolism in aging adults. Front Aging Neurosci 2017; 9:334Crossref, Medline, Google Scholar FiguresReferencesCited byDetailsCited bySerum complement proteins rather than inflammatory factors is effective in predicting psychosis in individuals at clinical high risk12 January 2023 | Translational Psychiatry, Vol. 13, No. 1Psychotic disorders as a framework for precision psychiatry6 March 2023 | Nature Reviews Neurology, Vol. 44Antipsychotic prescription, assumption and conversion to psychosis: resolving missing clinical links to optimize prevention through precision4 May 2022 | Schizophrenia, Vol. 8, No. 1Age‐related changes in self‐reported psychotic experiences in clinical help‐seeking population: From 15 to 45 years23 April 2022 | Early Intervention in Psychiatry, Vol. 16, No. 12Changes in inflammatory balance correlates with conversion to psychosis among individuals at clinical high-risk: A prospective cohort studyPsychiatry Research, Vol. 318Alterations in facial expressions in individuals at risk for psychosis: a facial electromyography approach using emotionally evocative film clips26 October 2022 | Psychological Medicine, Vol. 101Dynamic intervention-based biomarkers may reduce heterogeneity and motivate targeted interventions in clinical high risk for psychosisSchizophrenia Research, Vol. 246Eye movement indices as predictors of conversion to psychosis in individuals at clinical high risk20 July 2022 | European Archives of Psychiatry and Clinical Neuroscience, Vol. 12Further evidence that antipsychotic medication does not prevent long-term psychosis in higher-risk individuals18 September 2021 | European Archives of Psychiatry and Clinical Neuroscience, Vol. 272, No. 4Inefficient integration during multiple facial processing in pre-morbid and early phases of psychosis17 December 2021 | The World Journal of Biological Psychiatry, Vol. 23, No. 5Research Trends in Individuals at High Risk for Psychosis: A Bibliometric Analysis29 April 2022 | Frontiers in Psychiatry, Vol. 13Different patterns of association between white matter microstructure and plasma unsaturated fatty acids in those with high risk for psychosis and healthy participants13 April 2022 | General Psychiatry, Vol. 35, No. 2Attenuated niacin-induced skin flush response in individuals with clinical high risk for psychosis25 April 2022 | General Psychiatry, Vol. 35, No. 2Neurocognitive Assessments Are More Important Among Adolescents Than Adults for Predicting Psychosis in Clinical High RiskBiological Psychiatry: Cognitive Neuroscience and Neuroimaging, Vol. 7, No. 1Translating RDoC to real-world impact in developmental psychopathology: A neurodevelopmental framework for application of mental health risk calculators7 December 2021 | Development and Psychopathology, Vol. 33, No. 5Toward Generalizable and Transdiagnostic Tools for Psychosis Prediction: An Independent Validation and Improvement of the NAPLS-2 Risk Calculator in the Multisite PRONIA CohortBiological Psychiatry, Vol. 90, No. 9Prediction and Prevention in the Clinical High-Risk for Psychosis Paradigm: A Review of the Current Status and Recommendations for Future Directions of Inquiry22 October 2021 | Frontiers in Psychiatry, Vol. 12Psychiatric disorders in China: strengths and challenges of contemporary research and clinical services15 July 2021 | Psychological Medicine, Vol. 51, No. 12Preventive psychiatry: a blueprint for improving the mental health of young people18 May 2021 | World Psychiatry, Vol. 20, No. 2Individualized risk components guiding antipsychotic delivery in patients with a clinical high risk of psychosis: application of a risk calculator17 February 2021 | Psychological Medicine, Vol. 7Selection for psychosocial treatment for youth at clinical high risk for psychosis based on the North American Prodrome Longitudinal Study individualized risk calculator14 January 2020 | Early Intervention in Psychiatry, Vol. 15, No. 1Third external replication of an individualised transdiagnostic prediction model for the automatic detection of individuals at risk of psychosis using electronic health recordsSchizophrenia Research, Vol. 228Development of a probability calculator for psychosis risk in children, adolescents, and young adults12 January 2021 | Psychological Medicine, Vol. 2Frequency of Self-reported Psychotic Symptoms among 2542 Outpatients at Their First Visit for Mental Health Services6 January 2021 | Psychiatry, Vol. 172Counterpoint. Early intervention for psychosis risk syndromes: Minimizing risk and maximizing benefitSchizophrenia Research, Vol. 227Concordance and factor structure of subthreshold positive symptoms in youth at clinical high risk for psychosisSchizophrenia Research, Vol. 227Subtypes of Clinical High Risk for Psychosis that Predict Antipsychotic Effectiveness in Long-Term Remission12 October 2020 | Pharmacopsychiatry, Vol. 54, No. 01Prediction of psychosis: model development and internal validation of a personalized risk calculator14 December 2020 | Psychological Medicine, Vol. 19Cognitive dysfunction in a psychotropic medication-naïve, clinical high-risk sample from the ShangHai-At-Risk-for-Psychosis (SHARP) study: Associations with clinical outcomesSchizophrenia Research, Vol. 226Meta-analyzing the prevalence and prognostic effect of antipsychotic exposure in clinical high-risk (CHR): when things are not what they seem17 November 2020 | Psychological Medicine, Vol. 50, No. 16Transdiagnostic individualized clinically-based risk calculator for the automatic detection of individuals at-risk and the prediction of psychosis: external replication in 2,430,333 US patients29 October 2020 | Translational Psychiatry, Vol. 10, No. 1When to initiate antipsychotic treatment for psychotic symptoms: At the premorbid phase or first episode of psychosis?3 November 2020 | Australian & New Zealand Journal of Psychiatry, Vol. 26Sensorimotor and Activity Psychosis-Risk (SMAP-R) Scale: An Exploration of Scale Structure With Replication and Validation13 October 2020 | Schizophrenia Bulletin, Vol. 116Predicting psychosis risk using a specific measure of cognitive control: a 12-month longitudinal study11 September 2019 | Psychological Medicine, Vol. 50, No. 13Functional connectome organization predicts conversion to psychosis in clinical high-risk youth from the SHARP program8 November 2018 | Molecular Psychiatry, Vol. 25, No. 10The Current State of the Clinical High Risk for Psychosis Research ParadigmBiological Psychiatry, Vol. 88, No. 4Discovery and Validation of Prediction Algorithms for Psychosis in Youths at Clinical High RiskBiological Psychiatry: Cognitive Neuroscience and Neuroimaging, Vol. 5, No. 8Poor functional recovery is better predicted than conversion in studies of outcomes of clinical high risk of psychosis: insight from SHARP27 August 2019 | Psychological Medicine, Vol. 50, No. 9Real-world effectiveness of antipsychotic treatment in psychosis prevention in a 3-year cohort of 517 individuals at clinical high risk from the SHARP (ShangHai At Risk for Psychosis)21 May 2020 | Australian & New Zealand Journal of Psychiatry, Vol. 54, No. 7Clinical subtypes that predict conversion to psychosis: A canonical correlation analysis study from the ShangHai At Risk for Psychosis program5 September 2019 | Australian & New Zealand Journal of Psychiatry, Vol. 54, No. 5Brain functional connectivity data enhance prediction of clinical outcome in youth at risk for psychosisNeuroImage: Clinical, Vol. 26Calculating individualized risk components using a mobile app-based risk calculator for clinical high risk of psychosis: findings from ShangHai At Risk for Psychosis (SHARP) program16 December 2019 | Psychological Medicine, Vol. 26External validation and extension of the NAPLS-2 and SIPS-RC personalized risk calculators in an independent clinical high-risk samplePsychiatry Research, Vol. 279Schizophrenia Bulletin Volume 175Issue 9 September 01, 2018Pages 906-908 Metrics KeywordsUltra High RiskProdromal PsychosisRisk CalculatorTransitionPredictionDrs. Seidman and McCarley were founders and core members of the Shanghai At Risk for Psychosis (SHARP) project.PDF download History Accepted 18 June 2018 Published online 1 September 2018 Published in print 1 September 2018