Bridging the Gap: Cross-modal Knowledge Driven Network for Radiology Report Generation

Beichen Kang, Yao Zhang, Yun Xiong, Xing Jia, Jianbo Jiao, Ji Li

Research output: Chapter in Book/Report/Conference proceedingConference contribution

46 Downloads (Pure)

Abstract

Radiology report generation aims to generate medical reports based on given medical images, which can alleviate the workload of radiologists and has attracted significant research interest in recent years. However, existing studies have struggled to bridge the gap between the two different modalities (i.e. image and text) and generate clinically accurate reports. This is primarily due to the challenges in modelling the crossmodal mappings and the inefficiency of transferring knowledge across modalities. To address these challenges, in this paper, we propose to leverage a pre-constructed knowledge graph as a shared matrix that bridges the gap between visual and textual information, facilitating cross-modal knowledge transfer. This shared knowledge matrix effectively captures cross-modal mappings and aligns information between images and texts, thereby bridging the gap between modalities. Specifically, we propose a new module for knowledge distillation and preservation that integrates relevant knowledge representations into both visual and textual inputs, facilitating intuitive cross-modal knowledge interaction and enhancing the clinical accuracy of the generated reports. Experimental results on two benchmark datasets show the effectiveness of our method, outperforming state-of-the-arts in report generation.
Original languageEnglish
Title of host publication2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)
EditorsXingpeng Jiang, Haiying Wang, Reda Alhajj, Xiaohua Hu, Felix Engel, Mufti Mahmud, Nadia Pisanti, Xuefeng Cui, Hong Song
PublisherIEEE
Pages1202-1209
Number of pages8
ISBN (Electronic)9798350337488
ISBN (Print)9798350337495
DOIs
Publication statusPublished - 18 Jan 2024
Event2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) - Istanbul, Turkiye
Duration: 5 Dec 20238 Dec 2023

Publication series

NameIEEE International Conference on Bioinformatics and Biomedicine (BIBM)
PublisherIEEE
ISSN (Print)2156-1125
ISSN (Electronic)2156-1133

Conference

Conference2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)
Period5/12/238/12/23

Bibliographical note

Funding Information:
This work is partially supported by the National Key Research and Development Plan Project 2022YFC3600901, the Chinesisch-Deutsche Kooperationsgruppe: Precision Medicine in Pancreatic Cancer Antrag GZ 1456, and the Royal Society grant IES\R3\223050.

Publisher Copyright:
© 2023 IEEE.

Keywords

  • Visualization
  • Knowledge graphs
  • Radiology
  • Benchmark testing
  • Data mining
  • Bioinformatics
  • Knowledge transfer

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Computer Vision and Pattern Recognition
  • Automotive Engineering
  • Modelling and Simulation
  • Health Informatics

Fingerprint

Dive into the research topics of 'Bridging the Gap: Cross-modal Knowledge Driven Network for Radiology Report Generation'. Together they form a unique fingerprint.

Cite this