Where is my attention? An explainable AI exploration in water detection from SAR imagery

Abstract

Attention mechanisms have found extensive application in Deep Neural Networks (DNNs), with numerous experiments over time showcasing their efficacy in improving the overall performance of DNNs. Considering the black-box nature of DNNs, it is unclear how attention mechanisms affect model decision-making processes. Therefore, it is of great significance to explore the internal relationship between the focused area of attention mechanisms and the prediction results. For the first time, an explainable framework for the attention mechanism of Synthetic Aperture Radar (SAR) image analytics is proposed, integrating DNN and eXplainable Artificial Intelligence (XAI), which realizes the effective explanation for attention mechanisms across different locations. The framework consists of three parts: the water extraction network which contains the attention mechanism at different locations; the proposed Attention- Gradient Class Activation Mapping (A-GCAM) method for attribution analysis of attention mechanisms; and we invent Category-specific channel Score of Confidence Mapping (CSCM) to perform geo-visualization for the output features of attention mechanisms. Experiments are conducted with the widely used Sentinel-1 system for water detection, in which three attention mechanisms with different characteristics are added to the encoder and decoder in DNNs. The results show that the framework can make the decision-making process of attention mechanisms transparent, thus improving their comprehensiveness and trustworthiness in various tasks and providing a reliable approach to selecting suitable attention mechanisms for the given task.