Automated Radiology Report Generation Using Transformers

Abstract

Given the rapid increase of respiratory illnesses in recent times, the demand for medical report writing for chest X- Rays (CXR) has significantly increased. In practice, a specialized medical expert has to go through an X-Ray image to compile the accompanying report, which is tedious, not scalable, and potentially prone to human error. Therefore, automatic medical report generation (AMRG) solutions for CXR as a diagnostic assistance tool could play an important role in lowering the burden on radiologists, making them more productive. However, current AMRG solutions are still lagging far behind the performance of human experts due to the reasons such as the inability to extract the most relevant features to be used for the compilation of the report. We address this by proposing MERGIS: MEdical Report Generation using the Image Segmentation approach. MERGIS is a modern transformer-based encoder-decoder model that leverages image segmentation to improve the accuracy of automatic report generation. In this approach, the CXR images are segmented before feeding into the model, enabling the encoder to extract relevant visual features of the medical image resulting in more accurate radiography reports. The proposed model outperforms the current state-of-the-art model for report generation on the MIMIC-CXR dataset with performance scores: BLUE-1 = 0.296, METEOR = 0.128, ROUGE L = 0.335, and CIDEr = 1.150.