Learning from multiple annotators for medical image segmentation

Le Zhang; Ryutaro Tanno; Moucheng Xu; Yawen Huang; Kevin Bronik; Chen Jin; Joseph Jacob; Yefeng Zheng; Ling Shao; Olga Ciccarelli; Frederik Barkhof; Daniel Alexander

doi:10.1016/j.patcog.2023.109400

Learning from multiple annotators for medical image segmentation

Le Zhang, Ryutaro Tanno, Moucheng Xu, Yawen Huang, Kevin Bronik, Chen Jin, Joseph Jacob, Yefeng Zheng, Ling Shao, Olga Ciccarelli, Frederik Barkhof, Daniel Alexander

Engineering

Research output: Contribution to journal › Article › peer-review

2 Downloads (Pure)

Abstract

Supervised machine learning methods have been widely developed for segmentation tasks in recent years. However, the quality of labels has high impact on the predictive performance of these algorithms. This issue is particularly acute in the medical image domain, where both the cost of annotation and the inter-observer variability are high. Different human experts contribute estimates of the ”actual” segmentation labels in a typical label acquisition process, influenced by their personal biases and competency levels. The performance of automatic segmentation algorithms is limited when these noisy labels are used as the expert consensus label. In this work, we use two coupled CNNs to jointly learn, from purely noisy observations alone, the reliability of individual annotators and the expert consensus label distributions. The separation of the two is achieved by maximally describing the annotator’s “unreliable behavior” (we call it “maximally unreliable”) while achieving high fidelity with the noisy training data. We first create a toy segmentation dataset using MNIST and investigate the properties of the proposed algorithm. We then use three public medical imaging segmentation datasets to demonstrate our method’s efficacy, including both simulated (where necessary) and real-world annotations: 1) ISBI2015 (multiple-sclerosis lesions); 2) BraTS (brain tumors); 3) LIDC-IDRI (lung abnormalities). Finally, we create a real-world multiple sclerosis lesion dataset (QSMSC at UCL: Queen Square Multiple Sclerosis Center at UCL, UK) with manual segmentations from 4 different annotators (3 radiologists with different level skills and 1 expert to generate the expert consensus label). In all datasets, our method consistently outperforms competing methods and relevant baselines, especially when the number of annotations is small and the amount of disagreement is large. The studies also reveal that the system is capable of capturing the complicated spatial characteristics of annotators’ mistakes.

Original language	English
Article number	109400
Journal	Pattern Recognition
Volume	138
Early online date	11 Feb 2023
DOIs	https://doi.org/10.1016/j.patcog.2023.109400
Publication status	Published - Jun 2023

Bibliographical note

Acknowledgments
We would like to thank Swami Sankaranarayanan and Ardavan Saeedi at Butterfly Network for their feedback and initial discussions. We thank Parashkev Nachev, Nevin John, Olivia Goodkin, Jiaming Wu, Baris Kanber, Ferran Prados at Queen Square Institute of Neurology for analyzing and manually labeling the real-world MS data. MX is supported by GSK funding (BIDS3000034123) via UCL EPSRC CDT in i4health and UCL Engineering Dean’s Prize. PN is funded by the Wellcome Trust and JJ is supported by Wellcome Trust Clinical Research Career Development Fellowship: 209553/Z/17/Z. We are also grateful for EPSRC grants EP/R006032/1, EP/M020533/1, CRUK/EPSRC grant NS/A000069/1, and the NIHR UCLH Biomedical Research Centre, which support this work.

Access to Document

10.1016/j.patcog.2023.109400Licence: Creative Commons: Attribution (CC BY)

ZhangL2023LearningFinal published version, 4.53 MBLicence: Creative Commons: Attribution (CC BY)

Cite this

@article{c33c54a53d214148baf63ee16cf5b1cb,

title = "Learning from multiple annotators for medical image segmentation",

abstract = "Supervised machine learning methods have been widely developed for segmentation tasks in recent years. However, the quality of labels has high impact on the predictive performance of these algorithms. This issue is particularly acute in the medical image domain, where both the cost of annotation and the inter-observer variability are high. Different human experts contribute estimates of the ”actual” segmentation labels in a typical label acquisition process, influenced by their personal biases and competency levels. The performance of automatic segmentation algorithms is limited when these noisy labels are used as the expert consensus label. In this work, we use two coupled CNNs to jointly learn, from purely noisy observations alone, the reliability of individual annotators and the expert consensus label distributions. The separation of the two is achieved by maximally describing the annotator{\textquoteright}s “unreliable behavior” (we call it “maximally unreliable”) while achieving high fidelity with the noisy training data. We first create a toy segmentation dataset using MNIST and investigate the properties of the proposed algorithm. We then use three public medical imaging segmentation datasets to demonstrate our method{\textquoteright}s efficacy, including both simulated (where necessary) and real-world annotations: 1) ISBI2015 (multiple-sclerosis lesions); 2) BraTS (brain tumors); 3) LIDC-IDRI (lung abnormalities). Finally, we create a real-world multiple sclerosis lesion dataset (QSMSC at UCL: Queen Square Multiple Sclerosis Center at UCL, UK) with manual segmentations from 4 different annotators (3 radiologists with different level skills and 1 expert to generate the expert consensus label). In all datasets, our method consistently outperforms competing methods and relevant baselines, especially when the number of annotations is small and the amount of disagreement is large. The studies also reveal that the system is capable of capturing the complicated spatial characteristics of annotators{\textquoteright} mistakes.",

author = "Le Zhang and Ryutaro Tanno and Moucheng Xu and Yawen Huang and Kevin Bronik and Chen Jin and Joseph Jacob and Yefeng Zheng and Ling Shao and Olga Ciccarelli and Frederik Barkhof and Daniel Alexander",

note = "Acknowledgments We would like to thank Swami Sankaranarayanan and Ardavan Saeedi at Butterfly Network for their feedback and initial discussions. We thank Parashkev Nachev, Nevin John, Olivia Goodkin, Jiaming Wu, Baris Kanber, Ferran Prados at Queen Square Institute of Neurology for analyzing and manually labeling the real-world MS data. MX is supported by GSK funding (BIDS3000034123) via UCL EPSRC CDT in i4health and UCL Engineering Dean{\textquoteright}s Prize. PN is funded by the Wellcome Trust and JJ is supported by Wellcome Trust Clinical Research Career Development Fellowship: 209553/Z/17/Z. We are also grateful for EPSRC grants EP/R006032/1, EP/M020533/1, CRUK/EPSRC grant NS/A000069/1, and the NIHR UCLH Biomedical Research Centre, which support this work.",

year = "2023",

month = jun,

doi = "10.1016/j.patcog.2023.109400",

language = "English",

volume = "138",

journal = "Pattern Recognition",

issn = "0031-3203",

publisher = "Elsevier",

}

TY - JOUR

T1 - Learning from multiple annotators for medical image segmentation

AU - Zhang, Le

AU - Tanno, Ryutaro

AU - Xu, Moucheng

AU - Huang, Yawen

AU - Bronik, Kevin

AU - Jin, Chen

AU - Jacob, Joseph

AU - Zheng, Yefeng

AU - Shao, Ling

AU - Ciccarelli, Olga

AU - Barkhof, Frederik

AU - Alexander, Daniel

N1 - Acknowledgments We would like to thank Swami Sankaranarayanan and Ardavan Saeedi at Butterfly Network for their feedback and initial discussions. We thank Parashkev Nachev, Nevin John, Olivia Goodkin, Jiaming Wu, Baris Kanber, Ferran Prados at Queen Square Institute of Neurology for analyzing and manually labeling the real-world MS data. MX is supported by GSK funding (BIDS3000034123) via UCL EPSRC CDT in i4health and UCL Engineering Dean’s Prize. PN is funded by the Wellcome Trust and JJ is supported by Wellcome Trust Clinical Research Career Development Fellowship: 209553/Z/17/Z. We are also grateful for EPSRC grants EP/R006032/1, EP/M020533/1, CRUK/EPSRC grant NS/A000069/1, and the NIHR UCLH Biomedical Research Centre, which support this work.

PY - 2023/6

Y1 - 2023/6

N2 - Supervised machine learning methods have been widely developed for segmentation tasks in recent years. However, the quality of labels has high impact on the predictive performance of these algorithms. This issue is particularly acute in the medical image domain, where both the cost of annotation and the inter-observer variability are high. Different human experts contribute estimates of the ”actual” segmentation labels in a typical label acquisition process, influenced by their personal biases and competency levels. The performance of automatic segmentation algorithms is limited when these noisy labels are used as the expert consensus label. In this work, we use two coupled CNNs to jointly learn, from purely noisy observations alone, the reliability of individual annotators and the expert consensus label distributions. The separation of the two is achieved by maximally describing the annotator’s “unreliable behavior” (we call it “maximally unreliable”) while achieving high fidelity with the noisy training data. We first create a toy segmentation dataset using MNIST and investigate the properties of the proposed algorithm. We then use three public medical imaging segmentation datasets to demonstrate our method’s efficacy, including both simulated (where necessary) and real-world annotations: 1) ISBI2015 (multiple-sclerosis lesions); 2) BraTS (brain tumors); 3) LIDC-IDRI (lung abnormalities). Finally, we create a real-world multiple sclerosis lesion dataset (QSMSC at UCL: Queen Square Multiple Sclerosis Center at UCL, UK) with manual segmentations from 4 different annotators (3 radiologists with different level skills and 1 expert to generate the expert consensus label). In all datasets, our method consistently outperforms competing methods and relevant baselines, especially when the number of annotations is small and the amount of disagreement is large. The studies also reveal that the system is capable of capturing the complicated spatial characteristics of annotators’ mistakes.

AB - Supervised machine learning methods have been widely developed for segmentation tasks in recent years. However, the quality of labels has high impact on the predictive performance of these algorithms. This issue is particularly acute in the medical image domain, where both the cost of annotation and the inter-observer variability are high. Different human experts contribute estimates of the ”actual” segmentation labels in a typical label acquisition process, influenced by their personal biases and competency levels. The performance of automatic segmentation algorithms is limited when these noisy labels are used as the expert consensus label. In this work, we use two coupled CNNs to jointly learn, from purely noisy observations alone, the reliability of individual annotators and the expert consensus label distributions. The separation of the two is achieved by maximally describing the annotator’s “unreliable behavior” (we call it “maximally unreliable”) while achieving high fidelity with the noisy training data. We first create a toy segmentation dataset using MNIST and investigate the properties of the proposed algorithm. We then use three public medical imaging segmentation datasets to demonstrate our method’s efficacy, including both simulated (where necessary) and real-world annotations: 1) ISBI2015 (multiple-sclerosis lesions); 2) BraTS (brain tumors); 3) LIDC-IDRI (lung abnormalities). Finally, we create a real-world multiple sclerosis lesion dataset (QSMSC at UCL: Queen Square Multiple Sclerosis Center at UCL, UK) with manual segmentations from 4 different annotators (3 radiologists with different level skills and 1 expert to generate the expert consensus label). In all datasets, our method consistently outperforms competing methods and relevant baselines, especially when the number of annotations is small and the amount of disagreement is large. The studies also reveal that the system is capable of capturing the complicated spatial characteristics of annotators’ mistakes.

U2 - 10.1016/j.patcog.2023.109400

DO - 10.1016/j.patcog.2023.109400

M3 - Article

SN - 0031-3203

VL - 138

JO - Pattern Recognition

JF - Pattern Recognition

M1 - 109400

ER -

Learning from multiple annotators for medical image segmentation

Abstract

Bibliographical note

Access to Document

Fingerprint

Cite this