Unsupervised Semantic Segmentation with Contrastive Translation Coding

Abstract

AbstractThe proposed new method constructs a contrastive learning task through unsupervised image-to-image translation and indirectly extracts domain-invariant features by maximizing mutual information. Specifically, by reusing the semantic segmentation network for image-to-image translation, the probability space of the segmentation output is projected into the RGB space, thereby constructing positive and negative samples to maximize the mutual information between the input image and the semantic segmentation RGB projection map. Meanwhile, the output space of the target domain is transferred to the source domain, and then the robust domain-invariant semantic features are extracted. We develop two adversarial transfer methods and a three-stage training paradigm of pre-training, cross-domain transfer, and self-supervised training. Experiments on benchmarks demonstrate that this method is reasonable and feasible. Comprehensive ablation studies and analyses are also carried out to reveal the advantages and disadvantages of the two designed transfer schemes and the effect and significance of color-invariant semantic information for unsupervised semantic segmentation tasks. The contrastive learning task, which is constructed through unsupervised cross-domain image-to-image translation, provides a new insight for cross-domain transfer learning to overcome the problem of domain shift.KeywordsDomain shiftUnsupervised semantic segmentationUnsupervised image-to-image translationContrastive learning