Abstract
Radiology report generation aims to generate medical reports based on given medical images, which can alleviate the workload of radiologists and has attracted significant research interest in recent years. However, existing studies have struggled to bridge the gap between the two different modalities (i.e. image and text) and generate clinically accurate reports. This is primarily due to the challenges in modelling the crossmodal mappings and the inefficiency of transferring knowledge across modalities. To address these challenges, in this paper, we propose to leverage a pre-constructed knowledge graph as a shared matrix that bridges the gap between visual and textual information, facilitating cross-modal knowledge transfer. This shared knowledge matrix effectively captures cross-modal mappings and aligns information between images and texts, thereby bridging the gap between modalities. Specifically, we propose a new module for knowledge distillation and preservation that integrates relevant knowledge representations into both visual and textual inputs, facilitating intuitive cross-modal knowledge interaction and enhancing the clinical accuracy of the generated reports. Experimental results on two benchmark datasets show the effectiveness of our method, outperforming state-of-the-arts in report generation.
Original language | English |
---|---|
Title of host publication | 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) |
Editors | Xingpeng Jiang, Haiying Wang, Reda Alhajj, Xiaohua Hu, Felix Engel, Mufti Mahmud, Nadia Pisanti, Xuefeng Cui, Hong Song |
Publisher | IEEE |
Pages | 1202-1209 |
Number of pages | 8 |
ISBN (Electronic) | 9798350337488 |
ISBN (Print) | 9798350337495 |
DOIs | |
Publication status | Published - 18 Jan 2024 |
Event | 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) - Istanbul, Turkiye Duration: 5 Dec 2023 → 8 Dec 2023 |
Publication series
Name | IEEE International Conference on Bioinformatics and Biomedicine (BIBM) |
---|---|
Publisher | IEEE |
ISSN (Print) | 2156-1125 |
ISSN (Electronic) | 2156-1133 |
Conference
Conference | 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) |
---|---|
Period | 5/12/23 → 8/12/23 |
Bibliographical note
Funding Information:This work is partially supported by the National Key Research and Development Plan Project 2022YFC3600901, the Chinesisch-Deutsche Kooperationsgruppe: Precision Medicine in Pancreatic Cancer Antrag GZ 1456, and the Royal Society grant IES\R3\223050.
Publisher Copyright:
© 2023 IEEE.
Keywords
- Visualization
- Knowledge graphs
- Radiology
- Benchmark testing
- Data mining
- Bioinformatics
- Knowledge transfer
ASJC Scopus subject areas
- Artificial Intelligence
- Computer Science Applications
- Computer Vision and Pattern Recognition
- Automotive Engineering
- Modelling and Simulation
- Health Informatics