Adjustable patch and feature prior token-based transformer for weakly supervised semantic segmentation

Abstract

Weakly supervised semantic segmentation is a challenging task, utilizing only low-cost weak supervision to produce pixel-level predictions. Existing transformer-based methods for weakly supervised semantic segmentation have some limitations: (1) Fixing patch size might destroy the structured semantics, which is unfriendly to objects of different scales, and (2) Ignoring the prior features when using multi-head attention mechanisms might lead to inaccurate segmentation localization. To tackle these issues, we proposed an effective transformer framework coupled with the adjustable patch and prior feature tokens, termed as APFPformer, in which an adjustable patch module is developed to split the image according to the area of the salient object for preserving the structured semantics in patches. A prior feature token module is devised to exploit the edge and texture information as prior tokens, ensuring the gain of discriminative representation. Additionally, a single-stage scheme is applied to reduce the computation and rapid segmentation process. Our experiments demonstrate the superiority of our approach over early methods, gaining competitive mean Intersection-over-Union scores of 67.9% on the PASCAL VOC2012 dataset and 40.2% on the MS COCO dataset.