Semantic Image Segmentation by Dynamic Discriminative Prototypes

Abstract

Semantic segmentation achieves significant success through large-scale training data. Meanwhile, few-shot semantic segmentation was proposed to segment image regions of novel classes through few labeled testing data. However, it ignores classes previously learned from training data. This paper proposes a new segmentation framework called segmentation by dynamic prototype (SDP), which can simultaneously segment the image regions of base classes learned from many training data and novel classes learned from a few testing data. SDP performs segmentation by searching for the nearest prototype of each pixel's features. Different prototypes are representative features for different classes. In testing, SDP dynamically constructs novel prototypes according to support images while maintaining base prototypes learned from training images. The main challenge of SDP is how to achieve intra-image compactness, intra-class compactness, and inter-class separability simultaneously in the feature space. To tackle these challenges, we first introduce a discriminative pixelwise feature and prototype training method to improve the above three types of feature discriminability. We then introduce a mask refinement process in testing, which refines support images' masks to extract more separable novel prototypes. In addition, we introduce a prototype adaptation process in testing, which allows all prototypes to adapt to query images to reduce the prototypes' intra-class variances. Our approach achieves state-of-the-art results for novel class segmentation against existing few-shot semantic segmentation methods on PASCAL-5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> and COCO-20 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> benchmarks in combination with superior runtime efficiency. In addition, our method can maintain strong results in the segmentation of base classes.