Exploring more concentrated and consistent activation regions for cross-domain semantic segmentation

Abstract

The cross-domain semantic segmentation in urban scenes is challenging due to the gap between the source and the target domains. Although the existing methods achieve remarkable performance, these methods have some limitations. First, the region activated by the backbone network is dispersed. The dispersed feature activation region is difficult to help the feature distribution alignment between the two domains. Second, these methods ignore the inconsistent problem between the activation regions of the two domains. Intuitively, the consistent feature activation region between the two domains is beneficial to align the feature distribution. In this paper, we present a novel approach for cross-domain semantic segmentation. Specifically, we propose a kernel-based channel attention (KCA) module and a mutual information domain adaptation (MIDA) module. The KCA module models the relationship between channels by using explicit non-linear kernel mappings for generating KCA features that aim to obtain more concentrated activation regions of the category. The MIDA module is a strategy of the KCA feature distribution alignment between the two domains. Specifically, the MIDA module contains a mutual information loss function and two adversarial learning loss functions. The three loss functions work together to obtain more consistent activation regions of the KCA feature between the two domains. Finally, the aligned KCA feature is fused with the feature of backbone to guide the semantic segmentation. Furthermore, self-supervised learning and image translation methods are leveraged to learn a better cross-domain semantic segmentation model. Extensive experiments demonstrate that the proposed approach achieves competitive performance against the state-of-the-art methods on challenging synthetic-to-real tasks.