Large-scale external validation and comparison of prognostic models: An application to chronic obstructive pulmonary disease

Beniamino Guerra, Joan Soriano, sarah haile, Bernd Lamprecht, Ana Ramirez, Alice Turner, Pablo Martinez-Camblor, Bernhard Kaiser, Inmaculada Alfageme, Pere Almagro, Ciro Casanova, Juan Soler-Cataluna, Juan P de-Torres, Marc Miravitlles, Bartolome Celli, Gerben Ter Riet, Jose M Marin, Patricia Sobradillo, Cristina González-Estévez, Judith Garcia-AymerichBernard Langer, MeiLan K Han, Arnulf Langhammer, Linda Leivseth, Per Bakke, Toru Oga, JO Johanessen, Borja Cosio, Nicholas Roche, Julio Ancochea, Pierre-Régis Burgel, Don Sin, Milo Puhan

Research output: Contribution to journalArticlepeer-review

10 Citations (Scopus)
145 Downloads (Pure)


Background: External validations and comparisons of prognostic models or scores are a pre-requisite for their use in routine clinical care but lacking in most medical fields including chronic obstructive pulmonary disease (COPD). Our aim was to externally validate and concurrently compare prognostic scores for 3-year all-cause mortality in mostly multimorbid patients with COPD.

Methods: We relied on 24 cohort studies of the COPD Cohorts Collaborative International Assessment (3CIA) consortium, corresponding to primary, secondary and tertiary care in Europe, the Americas and Japan. They include globally 15762 patients with COPD (1871 deaths and 42203 person-years of follow-up). We used network meta-analysis adapted to multiple score comparison (MSC), following a frequentist two-stage approach; thus, we were able to compare all scores in a single analytical framework accounting for correlations among scores within cohorts. We assessed transitivity, heterogeneity and inconsistency and provided a performance ranking of the prognostic scores.

Results: Depending on data availability, between 2 and 9 prognostic scores could be calculated for each cohort. The BODE score (body mass index, airflow obstruction, dyspnea and exercise capacity) had a median AUC of 0.679 [1st quartile-3rd quartile = 0.655-0.733] across cohorts. The ADO score (age, dyspnea and airflow obstruction) showed the best performance for predicting mortality (difference AUCADO - AUCBODE = 0.015 [95% confidence interval (CI) = -0.002 to 0.032; p = 0.08) followed by the updated BODE (AUCBODE updated - AUCBODE = 0.008 [95% CI = -0.005 to +0.022; p = 0.23). The assumption of transitivity was not violated. Heterogeneity across direct comparisons was small and we did not identify any local or global inconsistency.

Conclusions: Our analyses showed best discriminatory performance for the ADO and updated BODE scores in patients with COPD. A limitation to be addressed in future studies is the extension of MSC network meta-analysis to measures of calibration. MSC network meta-analysis can be applied to prognostic scores in any the medical field to identify the best scores, possibly paving the way for stratified medicine, public health and research.
Original languageEnglish
Article number33
JournalBMC medicine
Publication statusPublished - 2 Mar 2018


  • performance comparison
  • network meta-analysis
  • COPD
  • prognostic scores
  • large-scale external validation


Dive into the research topics of 'Large-scale external validation and comparison of prognostic models: An application to chronic obstructive pulmonary disease'. Together they form a unique fingerprint.

Cite this