Token-word mixer meets object-aware transformer for referring image segmentation

Abstract

Referring image segmentation aims to generate a binary mask of the target object according to a referring expression. Some recent works argue that post-fusion paradigm may result in inconsistency and insufficiency issue and propose to integrate textual features during the visual encoding process. Although effective, they do fusion in a single way at each stage of encoder, e.g. utilizing cross attention mechanisms. This single fusion method ignores local and detailed image information correlated with language due to the incapability of attention in capturing high-frequencies information. To address this issue, we propose a Token-Word Mixer, which takes into consideration the characteristics of convolution and attention, and achieves more comprehensive interactions and alignments of multi-modal features through a mix operation. Furthermore, existing methods that rely solely on grid features lack perception of the target object and inference of relationships between objects, making it difficult to associate and align semantic information of target objects during multi-modal fusion when referring expressions or image scenes are complex. Therefore, we propose to incorporates object-level information by exploiting a DETR-based detector to provide region features, and the Object-Aware Transformer encoder with an additional learnable token is proposed to perceive effective information associated with the target object. Based on the enhanced cross-modal features and the aggregated token, we adopt query-based mask generation method instead of pixel classification framework for referring image segmentation. Extensive experiments and ablation studies indicate the effectiveness of our proposed methods.