Mining confident supervision by prototypes discovering and annotation selection for weakly supervised semantic segmentation

Abstract

Weakly supervised semantic segmentation methods generate proxy annotations for images based on image-level labels, and further optimize the segmentation network on these annotations with various constraints. However, the performance of current segmentation methods is often constrained by the quality of proxy annotations, even if saliency maps can work as strong background priors. Therefore, how to improve the quality of proxy annotations by mining more confident supervision has become a problem worth exploring and challenging. In this paper, we investigate two novel mechanisms to produce proxy annotations with higher quality: (1) designing a principal prototypical features discovering (PPFD) strategy for object localization, which is capable of utilizing image-level labels and coarse pixel-level annotations jointly, and (2) designing an effective annotation selection and refinement (ASR) strategy for improving annotations’ quality. The proposed strategy can localize foreground pixels by means of extracting prototypical feature from activation features of a group of images. Subsequently, the resulting PPFD based localization maps are utilized to generate proxy annotations for weakly supervised semantic segmentation. In addition, ASR is employed to recognize and refine the low-quality proxy annotations by self-judgment based mask scoring and discriminative regions mining. Experimental results on the PASCAL VOC 2012 and COCO 2014 datasets demonstrate that the proposed algorithm performs better than the state-of-the-art methods on the segmentation benchmark.