Evaluation of clinical prediction models (part 1): from development to external validation

Gary S Collins; Paula  Dhiman; Jie Ma; Michael M Schlussel; Lucinda Archer; Ben Van Calster; Frank E Harrell Jr; Glen P  Martin; Karel G M Moons; Maarten van Smeden; Matthew Sperrin; Garrett S Bullock; Richard D Riley

doi:10.1136/bmj-2023-074819

Evaluation of clinical prediction models (part 1): from development to external validation

Gary S Collins^*, Paula Dhiman, Jie Ma, Michael M Schlussel, Lucinda Archer, Ben Van Calster, Frank E Harrell Jr, Glen P Martin, Karel G M Moons, Maarten van Smeden, Matthew Sperrin, Garrett S Bullock, Richard D Riley

^*Corresponding author for this work

Applied Health Research

Research output: Contribution to journal › Article › peer-review

4 Downloads (Pure)

Abstract

Evaluating the performance of a clinical prediction model is crucial to establish its predictive accuracy in the populations and settings intended for use. In this article, the first in a three part series, Collins and colleagues describe the importance of a meaningful evaluation using internal, internal-external, and external validation, as well as exploring heterogeneity, fairness, and generalisability in model performance.

Original language	English
Article number	e074819
Number of pages	11
Journal	BMJ
Volume	384
DOIs	https://doi.org/10.1136/bmj-2023-074819
Publication status	Published - 8 Jan 2024

Bibliographical note

Funding:
This work was supported by Cancer Research UK (C49297/A27294, which supports GSC, JM, and MMS; and PRCPJT-Nov21\100021, which supports PD). The Medical Research Council Better Methods Better Research (grant MR/V038168/1, which supports GSC, LA, and RDR), the EPSRC (Engineering and Physical Sciences Research Council) grant for “Artificial intelligence innovation to accelerate health research” (EP/Y018516/1, which supports GSC, LA, PD, and RDR). National Institute for Health and Care Research Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham (which supports RDR), the Research Foundation-Flanders (G097322N, which supports BVC), Internal Funds KU Leuven (C24M/20/064, which supports BVC), National Center for Advancing Translational Sciences (Clinical Translational Science Award 5UL1TR002243-03, which supports FEH), National Institutes of Health (NHLBI 1OT2HL156812-01, which supports FEH), and the ACTIV Integration of Host-targeting Therapies for COVID-19 Administrative Coordinating Center from the National Heart, Lung, and Blood Institute (which supports FEH) The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

Access to Document

10.1136/bmj-2023-074819Licence: Creative Commons: Attribution (CC BY)

CollinsG2024EvaluationFinal published version, 1.09 MBLicence: Creative Commons: Attribution (CC BY)

https://www.bmj.com/content/bmj/384/bmj-2023-074819.full.pdfLicence: Creative Commons: Attribution (CC BY)

Sample Size guidance for developing and validating reliable and fair AI PREDICTion models in healthcare (SS-PREDICT)
Cazier, J., Riley, R., Snell, K., Archer, L., Nirantharakumar, K., Cazier, J., Ensor, J. & Denniston, A.
UK Research and Innovation
2/10/23 → 1/04/25
Project: Research Councils

Cite this

@article{176bf926ff1249b78172b9e7f298c0f6,

title = "Evaluation of clinical prediction models (part 1): from development to external validation",

abstract = "Evaluating the performance of a clinical prediction model is crucial to establish its predictive accuracy in the populations and settings intended for use. In this article, the first in a three part series, Collins and colleagues describe the importance of a meaningful evaluation using internal, internal-external, and external validation, as well as exploring heterogeneity, fairness, and generalisability in model performance.",

author = "Collins, {Gary S} and Paula Dhiman and Jie Ma and Schlussel, {Michael M} and Lucinda Archer and {Van Calster}, Ben and Jr, {Frank E Harrell} and Martin, {Glen P} and Moons, {Karel G M} and {van Smeden}, Maarten and Matthew Sperrin and Bullock, {Garrett S} and Riley, {Richard D}",

note = "Funding: This work was supported by Cancer Research UK (C49297/A27294, which supports GSC, JM, and MMS; and PRCPJT-Nov21\100021, which supports PD). The Medical Research Council Better Methods Better Research (grant MR/V038168/1, which supports GSC, LA, and RDR), the EPSRC (Engineering and Physical Sciences Research Council) grant for “Artificial intelligence innovation to accelerate health research” (EP/Y018516/1, which supports GSC, LA, PD, and RDR). National Institute for Health and Care Research Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham (which supports RDR), the Research Foundation-Flanders (G097322N, which supports BVC), Internal Funds KU Leuven (C24M/20/064, which supports BVC), National Center for Advancing Translational Sciences (Clinical Translational Science Award 5UL1TR002243-03, which supports FEH), National Institutes of Health (NHLBI 1OT2HL156812-01, which supports FEH), and the ACTIV Integration of Host-targeting Therapies for COVID-19 Administrative Coordinating Center from the National Heart, Lung, and Blood Institute (which supports FEH) The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.",

year = "2024",

month = jan,

day = "8",

doi = "10.1136/bmj-2023-074819",

language = "English",

volume = "384",

journal = "BMJ",

issn = "1756-1833",

publisher = "BMJ Publishing Group",

}

TY - JOUR

T1 - Evaluation of clinical prediction models (part 1)

T2 - from development to external validation

AU - Collins, Gary S

AU - Dhiman, Paula

AU - Ma, Jie

AU - Schlussel, Michael M

AU - Archer, Lucinda

AU - Van Calster, Ben

AU - Jr, Frank E Harrell

AU - Martin, Glen P

AU - Moons, Karel G M

AU - van Smeden, Maarten

AU - Sperrin, Matthew

AU - Bullock, Garrett S

AU - Riley, Richard D

N1 - Funding: This work was supported by Cancer Research UK (C49297/A27294, which supports GSC, JM, and MMS; and PRCPJT-Nov21\100021, which supports PD). The Medical Research Council Better Methods Better Research (grant MR/V038168/1, which supports GSC, LA, and RDR), the EPSRC (Engineering and Physical Sciences Research Council) grant for “Artificial intelligence innovation to accelerate health research” (EP/Y018516/1, which supports GSC, LA, PD, and RDR). National Institute for Health and Care Research Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham (which supports RDR), the Research Foundation-Flanders (G097322N, which supports BVC), Internal Funds KU Leuven (C24M/20/064, which supports BVC), National Center for Advancing Translational Sciences (Clinical Translational Science Award 5UL1TR002243-03, which supports FEH), National Institutes of Health (NHLBI 1OT2HL156812-01, which supports FEH), and the ACTIV Integration of Host-targeting Therapies for COVID-19 Administrative Coordinating Center from the National Heart, Lung, and Blood Institute (which supports FEH) The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

PY - 2024/1/8

Y1 - 2024/1/8

N2 - Evaluating the performance of a clinical prediction model is crucial to establish its predictive accuracy in the populations and settings intended for use. In this article, the first in a three part series, Collins and colleagues describe the importance of a meaningful evaluation using internal, internal-external, and external validation, as well as exploring heterogeneity, fairness, and generalisability in model performance.

AB - Evaluating the performance of a clinical prediction model is crucial to establish its predictive accuracy in the populations and settings intended for use. In this article, the first in a three part series, Collins and colleagues describe the importance of a meaningful evaluation using internal, internal-external, and external validation, as well as exploring heterogeneity, fairness, and generalisability in model performance.

U2 - 10.1136/bmj-2023-074819

DO - 10.1136/bmj-2023-074819

M3 - Article

SN - 1756-1833

VL - 384

JO - BMJ

JF - BMJ

M1 - e074819

ER -

Evaluation of clinical prediction models (part 1): from development to external validation

Abstract

Bibliographical note

Access to Document

Fingerprint

Projects

Sample Size guidance for developing and validating reliable and fair AI PREDICTion models in healthcare (SS-PREDICT)

Cite this