Evaluation of clinical prediction models (part 2): how to undertake an external validation study

Richard D Riley; Lucinda Archer; Kym I E Snell; Joie Ensor; Paula Dhiman; Glen P Martin; Laura J Bonnett; Gary S Collins

doi:10.1136/bmj-2023-074820

Evaluation of clinical prediction models (part 2): how to undertake an external validation study

Richard D Riley^*, Lucinda Archer, Kym I E Snell, Joie Ensor, Paula Dhiman, Glen P Martin, Laura J Bonnett, Gary S Collins

^*Corresponding author for this work

Applied Health Research

Research output: Contribution to journal › Article › peer-review

6 Downloads (Pure)

Abstract

External validation studies are an important but often neglected part of prediction model research. In this article, the second in a series on model evaluation, Riley and colleagues explain what an external validation study entails and describe the key steps involved, from establishing a high quality dataset to evaluating a model’s predictive performance and clinical usefulness.

Original language	English
Article number	e074820
Journal	BMJ (Clinical research ed.)
Volume	384
DOIs	https://doi.org/10.1136/bmj-2023-074820
Publication status	Published - 15 Jan 2024

Bibliographical note

Funding: This paper presents independent research supported (for RDR, LA, KIES, JE, PD, and GSC) by an Engineering and Physical Sciences Research Council (EPSRC) grant for “Artificial intelligence innovation to accelerate health research” (EP/Y018516/1); a National Institute for Health and Care Research (NIHR)-Medical Research Council (MRC) Better Methods Better Research grant (for RDR, LA, KIES, JE, and GSC) (MR/V038168/1), and the NIHR Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham (for RDR, LA, JE, and KIES). The views expressed are those of the author(s) and not necessarily those of the NHS, NIHR, or UK Department of Health and Social Care. GSC was also supported by Cancer Research UK (programme grant C49297/A27294). The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

Keywords

Humans
Models, Statistical
Prognosis

Access to Document

10.1136/bmj-2023-074820Licence: Creative Commons: Attribution (CC BY)

RileyRD2024EvaluationFinal published version, 1 MBLicence: Creative Commons: Attribution (CC BY)

Sample Size guidance for developing and validating reliable and fair AI PREDICTion models in healthcare (SS-PREDICT)
Cazier, J., Riley, R., Snell, K., Archer, L., Nirantharakumar, K., Cazier, J., Ensor, J. & Denniston, A.
UK Research and Innovation
2/10/23 → 1/04/25
Project: Research Councils

Cite this

@article{878f2a4041304209ad2ae6f4e6c938d2,

title = "Evaluation of clinical prediction models (part 2): how to undertake an external validation study",

abstract = "External validation studies are an important but often neglected part of prediction model research. In this article, the second in a series on model evaluation, Riley and colleagues explain what an external validation study entails and describe the key steps involved, from establishing a high quality dataset to evaluating a model{\textquoteright}s predictive performance and clinical usefulness.",

keywords = "Humans, Models, Statistical, Prognosis",

author = "Riley, {Richard D} and Lucinda Archer and Snell, {Kym I E} and Joie Ensor and Paula Dhiman and Martin, {Glen P} and Bonnett, {Laura J} and Collins, {Gary S}",

note = "Funding: This paper presents independent research supported (for RDR, LA, KIES, JE, PD, and GSC) by an Engineering and Physical Sciences Research Council (EPSRC) grant for “Artificial intelligence innovation to accelerate health research” (EP/Y018516/1); a National Institute for Health and Care Research (NIHR)-Medical Research Council (MRC) Better Methods Better Research grant (for RDR, LA, KIES, JE, and GSC) (MR/V038168/1), and the NIHR Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham (for RDR, LA, JE, and KIES). The views expressed are those of the author(s) and not necessarily those of the NHS, NIHR, or UK Department of Health and Social Care. GSC was also supported by Cancer Research UK (programme grant C49297/A27294). The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.",

year = "2024",

month = jan,

day = "15",

doi = "10.1136/bmj-2023-074820",

language = "English",

volume = "384",

journal = "BMJ (Clinical research ed.)",

issn = "1756-1833",

publisher = "BMJ Publishing Group",

}

TY - JOUR

T1 - Evaluation of clinical prediction models (part 2)

T2 - how to undertake an external validation study

AU - Riley, Richard D

AU - Archer, Lucinda

AU - Snell, Kym I E

AU - Ensor, Joie

AU - Dhiman, Paula

AU - Martin, Glen P

AU - Bonnett, Laura J

AU - Collins, Gary S

N1 - Funding: This paper presents independent research supported (for RDR, LA, KIES, JE, PD, and GSC) by an Engineering and Physical Sciences Research Council (EPSRC) grant for “Artificial intelligence innovation to accelerate health research” (EP/Y018516/1); a National Institute for Health and Care Research (NIHR)-Medical Research Council (MRC) Better Methods Better Research grant (for RDR, LA, KIES, JE, and GSC) (MR/V038168/1), and the NIHR Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham (for RDR, LA, JE, and KIES). The views expressed are those of the author(s) and not necessarily those of the NHS, NIHR, or UK Department of Health and Social Care. GSC was also supported by Cancer Research UK (programme grant C49297/A27294). The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

PY - 2024/1/15

Y1 - 2024/1/15

N2 - External validation studies are an important but often neglected part of prediction model research. In this article, the second in a series on model evaluation, Riley and colleagues explain what an external validation study entails and describe the key steps involved, from establishing a high quality dataset to evaluating a model’s predictive performance and clinical usefulness.

AB - External validation studies are an important but often neglected part of prediction model research. In this article, the second in a series on model evaluation, Riley and colleagues explain what an external validation study entails and describe the key steps involved, from establishing a high quality dataset to evaluating a model’s predictive performance and clinical usefulness.

KW - Humans

KW - Models, Statistical

KW - Prognosis

U2 - 10.1136/bmj-2023-074820

DO - 10.1136/bmj-2023-074820

M3 - Article

C2 - 38224968

SN - 1756-1833

VL - 384

JO - BMJ (Clinical research ed.)

JF - BMJ (Clinical research ed.)

M1 - e074820

ER -

Evaluation of clinical prediction models (part 2): how to undertake an external validation study

Abstract

Bibliographical note

Keywords

Access to Document

Fingerprint

Projects

Sample Size guidance for developing and validating reliable and fair AI PREDICTion models in healthcare (SS-PREDICT)

Cite this