Gradient-coupled cross-patch attention map for weakly supervised semantic segmentation

Abstract

Most advanced weakly supervised semantic segmentation studies have explored various strategies for refining the class activation mapping (CAM) maps and thus producing pixelwise pseudomasks for training segmentation networks. However, CAM-based approaches suffer from the limited receptive fields of convolutional neural networks (CNNs) when exploring long-range feature dependencies, resulting in incomplete object region detection and inferior segmentation performance. On the other hand, the self-attention mechan ism has proven to be effective in modeling global contexts, providing a potential perspective for producing integral activation maps. Nevertheless, the use of self-attention maps to generate pseudomasks faces significant obstacles, as the attention activations are class-agnostic, locally inconsistent and noisy. To address these issues, we propose a novel end-to-end transformer-based framework, namely the gradient-coupledcross-patchattentionmap (GC-PAM). First, the classification score gradients are backpropagated to recover the semantic knowledge of the attention map, which is then used to retrieve the object-relevant patches and object-irrelevant patches. Next, the corresponding object patch attention maps are aggregated through content-adaptive cross-patch coupling to construct comprehensive activation maps, whereas the object-irrelevant maps are employed to suppress the background noise. Subsequently, the resulting object activation maps are used to generate pseudolabels to supervise the segmentation branch. Extensive experiments are conducted to demonstrate the effectiveness of the proposed GC-PAM. Despite its simplicity and computational efficiency, the GC-PAM with the DeiT-S backbone surpasses the existing state-of-the-art approaches on the PASCAL VOC 2012 benchmark (75.3% val, 74.6% test). The results also demonstrate that the GC-PAM is a feasible alternative to CNN-based architectures.