Toward generalizable and transdiagnostic tools for psychosis prediction: an independent validation and improvement of the NAPLS-2 risk calculator in the multisite PRONIA cohort

Nikolaos Koutsouleris; Michelle Worthington; Dominic B. Dwyer; Lana Kambeitz-Ilankovic; Rachele Sanfelici; Paolo Fusar-Poli; Marlene Rosen; Stephan Ruhrmann; Alan Anticevic; Jean Addington; Diana O. Perkins; Carrie E. Bearden; Barbara A. Cornblatt; Kristin S. Cadenhead; Daniel H. Mathalon; Thomas McGlashan; Larry Seidman; Ming Tsuang; Elaine F. Walker; Scott W. Woods; Peter Falkai; Rebekka Lencer; Alessandro Bertolino; Joseph Kambeitz; Frauke Schultze-Lutter; Eva Meisenzahl; Raimo K.R. Salokangas; Jarmo Hietala; Paolo Brambilla; Rachel Upthegrove; Stefan Borgwardt; Stephen Wood; Raquel E. Gur; Philip McGuire; Tyrone D. Cannon

doi:10.1016/j.biopsych.2021.06.023

Toward generalizable and transdiagnostic tools for psychosis prediction: an independent validation and improvement of the NAPLS-2 risk calculator in the multisite PRONIA cohort

Nikolaos Koutsouleris^*, Michelle Worthington, Dominic B. Dwyer, Lana Kambeitz-Ilankovic, Rachele Sanfelici, Paolo Fusar-Poli, Marlene Rosen, Stephan Ruhrmann, Alan Anticevic, Jean Addington, Diana O. Perkins, Carrie E. Bearden, Barbara A. Cornblatt, Kristin S. Cadenhead, Daniel H. Mathalon, Thomas McGlashan, Larry Seidman, Ming Tsuang, Elaine F. Walker, Scott W. WoodsPeter Falkai, Rebekka Lencer, Alessandro Bertolino, Joseph Kambeitz, Frauke Schultze-Lutter, Eva Meisenzahl, Raimo K.R. Salokangas, Jarmo Hietala, Paolo Brambilla, Rachel Upthegrove, Stefan Borgwardt, Stephen Wood, Raquel E. Gur, Philip McGuire, Tyrone D. Cannon

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

4 Citations (Scopus)

Abstract

Background:

Transition to psychosis is among the most adverse outcomes of clinical high-risk (CHR) syndromes encompassing ultra-high risk (UHR) and basic symptom states. Clinical risk calculators may facilitate an early and individualized interception of psychosis, but their real-world implementation requires thorough validation across diverse risk populations, including young patients with depressive syndromes.

Methods:

We validated the previously described NAPLS-2 (North American Prodrome Longitudinal Study 2) calculator in 334 patients (26 with transition to psychosis) with CHR or recent-onset depression (ROD) drawn from the multisite European PRONIA (Personalised Prognostic Tools for Early Psychosis Management) study. Patients were categorized into three risk enrichment levels, ranging from UHR, over CHR, to a broad-risk population comprising patients with CHR or ROD (CHR|ROD). We assessed how risk enrichment and different predictive algorithms influenced prognostic performance using reciprocal external validation.

Results:

After calibration, the NAPLS-2 model predicted psychosis with a balanced accuracy (BAC) (sensitivity, specificity) of 68% (73%, 63%) in the PRONIA-UHR cohort, 67% (74%, 60%) in the CHR cohort, and 70% (73%, 66%) in patients with CHR|ROD. Multiple model derivation in PRONIA–CHR|ROD and validation in NAPLS-2–UHR patients confirmed that broader risk definitions produced more accurate risk calculators (CHR|ROD-based vs. UHR-based performance: 67% [68%, 66%] vs. 58% [61%, 56%]). Support vector machines were superior in CHR|ROD (BAC = 71%), while ridge logistic regression and support vector machines performed similarly in CHR (BAC = 67%) and UHR cohorts (BAC = 65%). Attenuated psychotic symptoms predicted psychosis across risk levels, while younger age and reduced processing speed became increasingly relevant for broader risk cohorts.

Conclusions:

Clinical-neurocognitive machine learning models operating in young patients with affective and CHR syndromes facilitate a more precise and generalizable prediction of psychosis. Future studies should investigate their therapeutic utility in large-scale clinical trials.

Original language	English
Pages (from-to)	632-642
Number of pages	11
Journal	Biological Psychiatry
Volume	90
Issue number	9
Early online date	6 Jul 2021
DOIs	https://doi.org/10.1016/j.biopsych.2021.06.023
Publication status	Published - 1 Nov 2021

Bibliographical note

Acknowledgments and Disclosures
PRONIA is a collaboration project funded by the European Union under the 7th Framework Programme under grant agreement no. 602152. NAPLS-2 was supported by the National Institutes of Health (NIH) (Grant Nos. U01 MH081902 [to TDC], P50 MH066286 [to CEB], U01 MH081857 [to BAC], U01 MH82022 [to SWW], U01 MH066134 [to JA], U01 MH081944 [to KSC], R01 U01 MH066069 [to DOP], R01 MH076989 [to DHM], U01 MH081928 [to LS], and U01 MH081988 [to EFW]). The HARMONY collaboration was supported by the NIH administrative supplement 3U01MH081928-07S1 (to LS). The funders were not involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

NK, LK-I, MR, SR, JA, DOP, CEB, BAC, KSC, DHM, TM, LS, MT, EFW, SWW, EM, FS-L, JK, RKRS, PB, SB, SW, and TDC were responsible for study design. NK, MW, LK-I, RS, MR, SR, JA, DOP, CEB, BAC, KSC, DHM, TM, LS, MT, EFW, SWW, EM, FS-L, JK, RKRS, PF, RL, AB, PB, RU, SB, SW, and TDC were responsible for data collection. NK, MW, and DBD were responsible for data analysis. NK, MW, PF-P, KSC, FS-L, PM, REG, and TDC were responsible for data interpretation. NK, MW, and TDC wrote the manuscript. DBD, LK-I, RS, PF-P, MR, SR, AA, JA, DOP, CEB, BAC, KSC, DHM, TM, MT, EFW, SWW, EM, FS-L, JK, RKRS, PF, RL, AB, PB, RU, SB, SW, PM, REG, and TDC reviewed the manuscript. NK takes the final responsibility for the decision to submit this work for publication.

We would like to thank Sen Dong, M.Sc., for the implementation of the models described in this manuscript in the NeuroMiner Model Libary available at www.proniapredictors.eu.

NK and EM hold an issued patent US20160192889A1 (“Adaptive pattern recognition for psychosis risk modelling”). All other authors report no biomedical financial interests or potential conflicts of interest.

Keywords

Clinical high-risk states
First-episode depression
Machine learning
Psychosis prediction
Reciprocal external validation
Risk calculators

ASJC Scopus subject areas

Biological Psychiatry

Access to Document

10.1016/j.biopsych.2021.06.023Licence: None: All rights reserved

Cite this

Koutsouleris, N., Worthington, M., Dwyer, D. B., Kambeitz-Ilankovic, L., Sanfelici, R., Fusar-Poli, P., Rosen, M., Ruhrmann, S., Anticevic, A., Addington, J., Perkins, D. O., Bearden, C. E., Cornblatt, B. A., Cadenhead, K. S., Mathalon, D. H., McGlashan, T., Seidman, L., Tsuang, M., Walker, E. F., ... Cannon, T. D. (2021). Toward generalizable and transdiagnostic tools for psychosis prediction: an independent validation and improvement of the NAPLS-2 risk calculator in the multisite PRONIA cohort. Biological Psychiatry, 90(9), 632-642. https://doi.org/10.1016/j.biopsych.2021.06.023

@article{0a4eccaeec7540fba14afd1de065cdba,

title = "Toward generalizable and transdiagnostic tools for psychosis prediction: an independent validation and improvement of the NAPLS-2 risk calculator in the multisite PRONIA cohort",

abstract = "Background: Transition to psychosis is among the most adverse outcomes of clinical high-risk (CHR) syndromes encompassing ultra-high risk (UHR) and basic symptom states. Clinical risk calculators may facilitate an early and individualized interception of psychosis, but their real-world implementation requires thorough validation across diverse risk populations, including young patients with depressive syndromes.Methods:We validated the previously described NAPLS-2 (North American Prodrome Longitudinal Study 2) calculator in 334 patients (26 with transition to psychosis) with CHR or recent-onset depression (ROD) drawn from the multisite European PRONIA (Personalised Prognostic Tools for Early Psychosis Management) study. Patients were categorized into three risk enrichment levels, ranging from UHR, over CHR, to a broad-risk population comprising patients with CHR or ROD (CHR|ROD). We assessed how risk enrichment and different predictive algorithms influenced prognostic performance using reciprocal external validation.Results:After calibration, the NAPLS-2 model predicted psychosis with a balanced accuracy (BAC) (sensitivity, specificity) of 68% (73%, 63%) in the PRONIA-UHR cohort, 67% (74%, 60%) in the CHR cohort, and 70% (73%, 66%) in patients with CHR|ROD. Multiple model derivation in PRONIA–CHR|ROD and validation in NAPLS-2–UHR patients confirmed that broader risk definitions produced more accurate risk calculators (CHR|ROD-based vs. UHR-based performance: 67% [68%, 66%] vs. 58% [61%, 56%]). Support vector machines were superior in CHR|ROD (BAC = 71%), while ridge logistic regression and support vector machines performed similarly in CHR (BAC = 67%) and UHR cohorts (BAC = 65%). Attenuated psychotic symptoms predicted psychosis across risk levels, while younger age and reduced processing speed became increasingly relevant for broader risk cohorts.Conclusions: Clinical-neurocognitive machine learning models operating in young patients with affective and CHR syndromes facilitate a more precise and generalizable prediction of psychosis. Future studies should investigate their therapeutic utility in large-scale clinical trials.",

keywords = "Clinical high-risk states, First-episode depression, Machine learning, Psychosis prediction, Reciprocal external validation, Risk calculators",

author = "Nikolaos Koutsouleris and Michelle Worthington and Dwyer, {Dominic B.} and Lana Kambeitz-Ilankovic and Rachele Sanfelici and Paolo Fusar-Poli and Marlene Rosen and Stephan Ruhrmann and Alan Anticevic and Jean Addington and Perkins, {Diana O.} and Bearden, {Carrie E.} and Cornblatt, {Barbara A.} and Cadenhead, {Kristin S.} and Mathalon, {Daniel H.} and Thomas McGlashan and Larry Seidman and Ming Tsuang and Walker, {Elaine F.} and Woods, {Scott W.} and Peter Falkai and Rebekka Lencer and Alessandro Bertolino and Joseph Kambeitz and Frauke Schultze-Lutter and Eva Meisenzahl and Salokangas, {Raimo K.R.} and Jarmo Hietala and Paolo Brambilla and Rachel Upthegrove and Stefan Borgwardt and Stephen Wood and Gur, {Raquel E.} and Philip McGuire and Cannon, {Tyrone D.}",

note = "Acknowledgments and Disclosures PRONIA is a collaboration project funded by the European Union under the 7th Framework Programme under grant agreement no. 602152. NAPLS-2 was supported by the National Institutes of Health (NIH) (Grant Nos. U01 MH081902 [to TDC], P50 MH066286 [to CEB], U01 MH081857 [to BAC], U01 MH82022 [to SWW], U01 MH066134 [to JA], U01 MH081944 [to KSC], R01 U01 MH066069 [to DOP], R01 MH076989 [to DHM], U01 MH081928 [to LS], and U01 MH081988 [to EFW]). The HARMONY collaboration was supported by the NIH administrative supplement 3U01MH081928-07S1 (to LS). The funders were not involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. NK, LK-I, MR, SR, JA, DOP, CEB, BAC, KSC, DHM, TM, LS, MT, EFW, SWW, EM, FS-L, JK, RKRS, PB, SB, SW, and TDC were responsible for study design. NK, MW, LK-I, RS, MR, SR, JA, DOP, CEB, BAC, KSC, DHM, TM, LS, MT, EFW, SWW, EM, FS-L, JK, RKRS, PF, RL, AB, PB, RU, SB, SW, and TDC were responsible for data collection. NK, MW, and DBD were responsible for data analysis. NK, MW, PF-P, KSC, FS-L, PM, REG, and TDC were responsible for data interpretation. NK, MW, and TDC wrote the manuscript. DBD, LK-I, RS, PF-P, MR, SR, AA, JA, DOP, CEB, BAC, KSC, DHM, TM, MT, EFW, SWW, EM, FS-L, JK, RKRS, PF, RL, AB, PB, RU, SB, SW, PM, REG, and TDC reviewed the manuscript. NK takes the final responsibility for the decision to submit this work for publication. We would like to thank Sen Dong, M.Sc., for the implementation of the models described in this manuscript in the NeuroMiner Model Libary available at www.proniapredictors.eu. NK and EM hold an issued patent US20160192889A1 (“Adaptive pattern recognition for psychosis risk modelling”). All other authors report no biomedical financial interests or potential conflicts of interest.",

year = "2021",

month = nov,

day = "1",

doi = "10.1016/j.biopsych.2021.06.023",

language = "English",

volume = "90",

pages = "632--642",

journal = "Biological Psychiatry",

issn = "0006-3223",

publisher = "Elsevier",

number = "9",

}

Koutsouleris, N, Worthington, M, Dwyer, DB, Kambeitz-Ilankovic, L, Sanfelici, R, Fusar-Poli, P, Rosen, M, Ruhrmann, S, Anticevic, A, Addington, J, Perkins, DO, Bearden, CE, Cornblatt, BA, Cadenhead, KS, Mathalon, DH, McGlashan, T, Seidman, L, Tsuang, M, Walker, EF, Woods, SW, Falkai, P, Lencer, R, Bertolino, A, Kambeitz, J, Schultze-Lutter, F, Meisenzahl, E, Salokangas, RKR, Hietala, J, Brambilla, P, Upthegrove, R, Borgwardt, S, Wood, S, Gur, RE, McGuire, P & Cannon, TD 2021, 'Toward generalizable and transdiagnostic tools for psychosis prediction: an independent validation and improvement of the NAPLS-2 risk calculator in the multisite PRONIA cohort', Biological Psychiatry, vol. 90, no. 9, pp. 632-642. https://doi.org/10.1016/j.biopsych.2021.06.023

Toward generalizable and transdiagnostic tools for psychosis prediction: an independent validation and improvement of the NAPLS-2 risk calculator in the multisite PRONIA cohort. / Koutsouleris, Nikolaos; Worthington, Michelle; Dwyer, Dominic B. et al.
In: Biological Psychiatry, Vol. 90, No. 9, 01.11.2021, p. 632-642.

Research output: Contribution to journal › Article › peer-review

TY - JOUR

T1 - Toward generalizable and transdiagnostic tools for psychosis prediction

T2 - an independent validation and improvement of the NAPLS-2 risk calculator in the multisite PRONIA cohort

AU - Koutsouleris, Nikolaos

AU - Worthington, Michelle

AU - Dwyer, Dominic B.

AU - Kambeitz-Ilankovic, Lana

AU - Sanfelici, Rachele

AU - Fusar-Poli, Paolo

AU - Rosen, Marlene

AU - Ruhrmann, Stephan

AU - Anticevic, Alan

AU - Addington, Jean

AU - Perkins, Diana O.

AU - Bearden, Carrie E.

AU - Cornblatt, Barbara A.

AU - Cadenhead, Kristin S.

AU - Mathalon, Daniel H.

AU - McGlashan, Thomas

AU - Seidman, Larry

AU - Tsuang, Ming

AU - Walker, Elaine F.

AU - Woods, Scott W.

AU - Falkai, Peter

AU - Lencer, Rebekka

AU - Bertolino, Alessandro

AU - Kambeitz, Joseph

AU - Schultze-Lutter, Frauke

AU - Meisenzahl, Eva

AU - Salokangas, Raimo K.R.

AU - Hietala, Jarmo

AU - Brambilla, Paolo

AU - Upthegrove, Rachel

AU - Borgwardt, Stefan

AU - Wood, Stephen

AU - Gur, Raquel E.

AU - McGuire, Philip

AU - Cannon, Tyrone D.

N1 - Acknowledgments and Disclosures PRONIA is a collaboration project funded by the European Union under the 7th Framework Programme under grant agreement no. 602152. NAPLS-2 was supported by the National Institutes of Health (NIH) (Grant Nos. U01 MH081902 [to TDC], P50 MH066286 [to CEB], U01 MH081857 [to BAC], U01 MH82022 [to SWW], U01 MH066134 [to JA], U01 MH081944 [to KSC], R01 U01 MH066069 [to DOP], R01 MH076989 [to DHM], U01 MH081928 [to LS], and U01 MH081988 [to EFW]). The HARMONY collaboration was supported by the NIH administrative supplement 3U01MH081928-07S1 (to LS). The funders were not involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. NK, LK-I, MR, SR, JA, DOP, CEB, BAC, KSC, DHM, TM, LS, MT, EFW, SWW, EM, FS-L, JK, RKRS, PB, SB, SW, and TDC were responsible for study design. NK, MW, LK-I, RS, MR, SR, JA, DOP, CEB, BAC, KSC, DHM, TM, LS, MT, EFW, SWW, EM, FS-L, JK, RKRS, PF, RL, AB, PB, RU, SB, SW, and TDC were responsible for data collection. NK, MW, and DBD were responsible for data analysis. NK, MW, PF-P, KSC, FS-L, PM, REG, and TDC were responsible for data interpretation. NK, MW, and TDC wrote the manuscript. DBD, LK-I, RS, PF-P, MR, SR, AA, JA, DOP, CEB, BAC, KSC, DHM, TM, MT, EFW, SWW, EM, FS-L, JK, RKRS, PF, RL, AB, PB, RU, SB, SW, PM, REG, and TDC reviewed the manuscript. NK takes the final responsibility for the decision to submit this work for publication. We would like to thank Sen Dong, M.Sc., for the implementation of the models described in this manuscript in the NeuroMiner Model Libary available at www.proniapredictors.eu. NK and EM hold an issued patent US20160192889A1 (“Adaptive pattern recognition for psychosis risk modelling”). All other authors report no biomedical financial interests or potential conflicts of interest.

PY - 2021/11/1

Y1 - 2021/11/1

N2 - Background: Transition to psychosis is among the most adverse outcomes of clinical high-risk (CHR) syndromes encompassing ultra-high risk (UHR) and basic symptom states. Clinical risk calculators may facilitate an early and individualized interception of psychosis, but their real-world implementation requires thorough validation across diverse risk populations, including young patients with depressive syndromes.Methods:We validated the previously described NAPLS-2 (North American Prodrome Longitudinal Study 2) calculator in 334 patients (26 with transition to psychosis) with CHR or recent-onset depression (ROD) drawn from the multisite European PRONIA (Personalised Prognostic Tools for Early Psychosis Management) study. Patients were categorized into three risk enrichment levels, ranging from UHR, over CHR, to a broad-risk population comprising patients with CHR or ROD (CHR|ROD). We assessed how risk enrichment and different predictive algorithms influenced prognostic performance using reciprocal external validation.Results:After calibration, the NAPLS-2 model predicted psychosis with a balanced accuracy (BAC) (sensitivity, specificity) of 68% (73%, 63%) in the PRONIA-UHR cohort, 67% (74%, 60%) in the CHR cohort, and 70% (73%, 66%) in patients with CHR|ROD. Multiple model derivation in PRONIA–CHR|ROD and validation in NAPLS-2–UHR patients confirmed that broader risk definitions produced more accurate risk calculators (CHR|ROD-based vs. UHR-based performance: 67% [68%, 66%] vs. 58% [61%, 56%]). Support vector machines were superior in CHR|ROD (BAC = 71%), while ridge logistic regression and support vector machines performed similarly in CHR (BAC = 67%) and UHR cohorts (BAC = 65%). Attenuated psychotic symptoms predicted psychosis across risk levels, while younger age and reduced processing speed became increasingly relevant for broader risk cohorts.Conclusions: Clinical-neurocognitive machine learning models operating in young patients with affective and CHR syndromes facilitate a more precise and generalizable prediction of psychosis. Future studies should investigate their therapeutic utility in large-scale clinical trials.

AB - Background: Transition to psychosis is among the most adverse outcomes of clinical high-risk (CHR) syndromes encompassing ultra-high risk (UHR) and basic symptom states. Clinical risk calculators may facilitate an early and individualized interception of psychosis, but their real-world implementation requires thorough validation across diverse risk populations, including young patients with depressive syndromes.Methods:We validated the previously described NAPLS-2 (North American Prodrome Longitudinal Study 2) calculator in 334 patients (26 with transition to psychosis) with CHR or recent-onset depression (ROD) drawn from the multisite European PRONIA (Personalised Prognostic Tools for Early Psychosis Management) study. Patients were categorized into three risk enrichment levels, ranging from UHR, over CHR, to a broad-risk population comprising patients with CHR or ROD (CHR|ROD). We assessed how risk enrichment and different predictive algorithms influenced prognostic performance using reciprocal external validation.Results:After calibration, the NAPLS-2 model predicted psychosis with a balanced accuracy (BAC) (sensitivity, specificity) of 68% (73%, 63%) in the PRONIA-UHR cohort, 67% (74%, 60%) in the CHR cohort, and 70% (73%, 66%) in patients with CHR|ROD. Multiple model derivation in PRONIA–CHR|ROD and validation in NAPLS-2–UHR patients confirmed that broader risk definitions produced more accurate risk calculators (CHR|ROD-based vs. UHR-based performance: 67% [68%, 66%] vs. 58% [61%, 56%]). Support vector machines were superior in CHR|ROD (BAC = 71%), while ridge logistic regression and support vector machines performed similarly in CHR (BAC = 67%) and UHR cohorts (BAC = 65%). Attenuated psychotic symptoms predicted psychosis across risk levels, while younger age and reduced processing speed became increasingly relevant for broader risk cohorts.Conclusions: Clinical-neurocognitive machine learning models operating in young patients with affective and CHR syndromes facilitate a more precise and generalizable prediction of psychosis. Future studies should investigate their therapeutic utility in large-scale clinical trials.

KW - Clinical high-risk states

KW - First-episode depression

KW - Machine learning

KW - Psychosis prediction

KW - Reciprocal external validation

KW - Risk calculators

UR - http://www.scopus.com/inward/record.url?scp=85114259062&partnerID=8YFLogxK

UR - https://kclpure.kcl.ac.uk/portal/en/publications/toward-generalizable-and-transdiagnostic-tools-for-psychosis-prediction(a24c02a1-c998-4743-bdf7-d861d8c8c841).html

U2 - 10.1016/j.biopsych.2021.06.023

DO - 10.1016/j.biopsych.2021.06.023

M3 - Article

C2 - 34482951

AN - SCOPUS:85114259062

SN - 0006-3223

VL - 90

SP - 632

EP - 642

JO - Biological Psychiatry

JF - Biological Psychiatry

IS - 9

ER -

Koutsouleris N, Worthington M, Dwyer DB, Kambeitz-Ilankovic L, Sanfelici R, Fusar-Poli P et al. Toward generalizable and transdiagnostic tools for psychosis prediction: an independent validation and improvement of the NAPLS-2 risk calculator in the multisite PRONIA cohort. Biological Psychiatry. 2021 Nov 1;90(9):632-642. Epub 2021 Jul 6. doi: 10.1016/j.biopsych.2021.06.023

Toward generalizable and transdiagnostic tools for psychosis prediction: an independent validation and improvement of the NAPLS-2 risk calculator in the multisite PRONIA cohort

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this