Automatic Report Generation for Chest X-Ray Images: A Multilevel Multi-attention Approach

Abstract

A comprehensive X-ray imaging report greatly assists the medical professional to investigate an indispensable condition and medication. The preparation of an extensive and diversified medical report by analysing the chest X-ray image is a time-consuming task and requires highly experienced professionals. This work targets the fundamental problem of generating a long and multifarious medical report for the chest X-ray image. It introduces a novel Multilevel Multi-Attention based encoder-decoder approach by combining Context Level Visual Attention and Textual Attention to generate a plausible medical report for different views of chest X-ray images. It exploited the proven ability of the Convolutional Neural Network to acquire course information of visual-spatial regions as an encoder. It leverages the strength of the Long Short-Term Memory network to learn long sequential dependencies and the ability of attention to focus on the prominent section as a decoder. The proposed method emphasizes on contextual coherence in intra and inter-sentence dependency within a report to improve the overall medical report generation quality. The effectiveness of the proposed model is evaluated on the publicly available IU chest X-ray dataset consisting of chest images along with multifarious radiology reports. The final performance of the proposed model is reported using the COCO-caption evaluation API. It shows a significant improvement in a medical report generation task compared to state-of-the-art methods.