Multi-attentional causal intervention networks for medical image diagnosis

Abstract

Medical image diagnosis has developed rapidly under the impetus of the deep network. Previous works mainly focus on improving the diagnostic accuracy of models, i.e., first use a backbone network to extract image global features and then feed it into the classifier for diagnosis. However, these methods do not fully explore the transparent and reasonable decision-making process of the final classification results, which is crucial for medical diagnosis. In this paper, we propose a framework called Causal Intervention-based Multi-head Attention network (CaIMA) to enhance the explainability of medical diagnosis from a causal inference perspective, by exploring the inherent causal relationship between multi-region attention and diagnosis results. Specifically, it consists of three key components: (1) The multi-region attention module enables the network to focus on the distinct discriminative lesion regions that hold causal relationships with the predicted outcome. (2) The attention-driven data augmentation module provides accurate localization of discriminative regions and enhances model explainability. (3) The causal intervention module aims to explore the intrinsic causal relationship between the attention map and the predicted outcome, encouraging the network to learn more useful attention maps for medical image diagnosis. Besides, to address the learning difficulty of this network, we further introduce a non-overlapping multiple attentional guidance loss that encourages the learned multiple attention maps to focus on specific lesion regions without overlapping. We compare the proposed CaIMA with state-of-the-art methods on multimedia medical datasets, including three public medical image datasets (Kvasir, ISIC2018, COVID-19) and one private dataset (CLC), and the experimental results substantiate the effectiveness of CaIMA in terms of diagnosis accuracy and explainability.