Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study

Richard D Riley; Kym I E Snell; Lucinda Archer; Joie Ensor; Thomas P A Debray; Ben Van Calster; Maarten Van Smeden; Gary S Collins

doi:10.1136/bmj-2023-074821

Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study

Richard D Riley^*, Kym I E Snell, Lucinda Archer, Joie Ensor, Thomas P A Debray, Ben Van Calster, Maarten Van Smeden, Gary S Collins

^*Corresponding author for this work

Applied Health Research

Research output: Contribution to journal › Article › peer-review

16 Downloads (Pure)

Abstract

An external validation study evaluates the performance of a prediction model in new data, but many of these studies are too small to provide reliable answers. In the third article of their series on model evaluation, Riley and colleagues describe how to calculate the sample size required for external validation studies, and propose to avoid rules of thumb by tailoring calculations to the model and setting at hand.

Original language	English
Article number	e074821
Journal	BMJ
Volume	384
DOIs	https://doi.org/10.1136/bmj-2023-074821
Publication status	Published - 22 Jan 2024

Bibliographical note

Funding: This paper presents independent research supported (for RDR, LA, KIES, JE, and GSC) by an Engineering and Physical Sciences Research Council (EPSRC) grant for “Artificial intelligence innovation to accelerate health research” (EP/Y018516/1); a Medical Research Council (MRC)-National Institute for Health and Care Research (NIHR) Better Methods Better Research grant (reference MR/V038168/1); and (for RDR, JE, KIES, and LA) the NIHR Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. The views expressed are those of the authors and not necessarily those of the NHS, NIHR or Department of Health and Social Care. GSC was also supported by Cancer Research UK (programme grant C49297/A27294). BVC was supported by the Research Foundation Flanders (FWO) (grant G097322N) and Internal Funds KU Leuven (grant C24M/20/064). The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

Access to Document

10.1136/bmj-2023-074821Licence: Creative Commons: Attribution (CC BY)

RileyRD2024EvaluationFinal published version, 953 KBLicence: Creative Commons: Attribution (CC BY)

https://www.bmj.com/lookup/doi/10.1136/bmj-2023-074821

Sample Size guidance for developing and validating reliable and fair AI PREDICTion models in healthcare (SS-PREDICT)
Cazier, J., Riley, R., Snell, K., Archer, L., Nirantharakumar, K., Cazier, J., Ensor, J. & Denniston, A.
UK Research and Innovation
2/10/23 → 1/04/25
Project: Research Councils

Cite this

@article{190aae8f4020409f9483fb6c3f35c2e5,

title = "Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study",

abstract = "An external validation study evaluates the performance of a prediction model in new data, but many of these studies are too small to provide reliable answers. In the third article of their series on model evaluation, Riley and colleagues describe how to calculate the sample size required for external validation studies, and propose to avoid rules of thumb by tailoring calculations to the model and setting at hand.",

author = "Riley, {Richard D} and Snell, {Kym I E} and Lucinda Archer and Joie Ensor and Debray, {Thomas P A} and {Van Calster}, Ben and {Van Smeden}, Maarten and Collins, {Gary S}",

note = "Funding: This paper presents independent research supported (for RDR, LA, KIES, JE, and GSC) by an Engineering and Physical Sciences Research Council (EPSRC) grant for “Artificial intelligence innovation to accelerate health research” (EP/Y018516/1); a Medical Research Council (MRC)-National Institute for Health and Care Research (NIHR) Better Methods Better Research grant (reference MR/V038168/1); and (for RDR, JE, KIES, and LA) the NIHR Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. The views expressed are those of the authors and not necessarily those of the NHS, NIHR or Department of Health and Social Care. GSC was also supported by Cancer Research UK (programme grant C49297/A27294). BVC was supported by the Research Foundation Flanders (FWO) (grant G097322N) and Internal Funds KU Leuven (grant C24M/20/064). The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.",

year = "2024",

month = jan,

day = "22",

doi = "10.1136/bmj-2023-074821",

language = "English",

volume = "384",

journal = "BMJ",

issn = "1756-1833",

publisher = "BMJ Publishing Group",

}

TY - JOUR

T1 - Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study

AU - Riley, Richard D

AU - Snell, Kym I E

AU - Archer, Lucinda

AU - Ensor, Joie

AU - Debray, Thomas P A

AU - Van Calster, Ben

AU - Van Smeden, Maarten

AU - Collins, Gary S

N1 - Funding: This paper presents independent research supported (for RDR, LA, KIES, JE, and GSC) by an Engineering and Physical Sciences Research Council (EPSRC) grant for “Artificial intelligence innovation to accelerate health research” (EP/Y018516/1); a Medical Research Council (MRC)-National Institute for Health and Care Research (NIHR) Better Methods Better Research grant (reference MR/V038168/1); and (for RDR, JE, KIES, and LA) the NIHR Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. The views expressed are those of the authors and not necessarily those of the NHS, NIHR or Department of Health and Social Care. GSC was also supported by Cancer Research UK (programme grant C49297/A27294). BVC was supported by the Research Foundation Flanders (FWO) (grant G097322N) and Internal Funds KU Leuven (grant C24M/20/064). The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

PY - 2024/1/22

Y1 - 2024/1/22

N2 - An external validation study evaluates the performance of a prediction model in new data, but many of these studies are too small to provide reliable answers. In the third article of their series on model evaluation, Riley and colleagues describe how to calculate the sample size required for external validation studies, and propose to avoid rules of thumb by tailoring calculations to the model and setting at hand.

AB - An external validation study evaluates the performance of a prediction model in new data, but many of these studies are too small to provide reliable answers. In the third article of their series on model evaluation, Riley and colleagues describe how to calculate the sample size required for external validation studies, and propose to avoid rules of thumb by tailoring calculations to the model and setting at hand.

U2 - 10.1136/bmj-2023-074821

DO - 10.1136/bmj-2023-074821

M3 - Article

SN - 1756-1833

VL - 384

JO - BMJ

JF - BMJ

M1 - e074821

ER -

Evaluation of clinical prediction models (part 3): calculating the sample size required for an external validation study

Abstract

Bibliographical note

Access to Document

Fingerprint

Projects

Sample Size guidance for developing and validating reliable and fair AI PREDICTion models in healthcare (SS-PREDICT)

Cite this