Semantic Segmentation Refinement Using Entropy and Boundary-guided Monte Carlo Sampling and Directed Regional Search

Abstract

Semantic segmentation requires both a large receptive field and accurate spatial information. Although existing methods based on a fully convolutional network have greatly improved the accuracy, the prediction results still do not show satisfactory when parsing small objects and boundary regions. We propose a refinement algorithm to improve the result generated by the front network. Our method takes a modified double-branches network to generate both segmentation masks and semantic boundaries, which serve as refinement algorithms' input. We creatively introduce information entropy to represent the confidence of the neural network's prediction corresponding to each pixel. The information entropy combined with the semantic boundary can capture those unpredictable pixels with low-confidence through Monte Carlo sampling. Each selected pixel will serve as the initial seed for directed local search and refinement. According to the initial seed, our purpose is tantamount to searching the neighbor high-confidence regions, and the re-labeling approach is based on high-confidence results. Remarkably, our method adopts a directed regional search strategy based on gradient descent to find the high-confidence region effectively. Our method can be flexibly embedded into the existing encoder backbone at a trivial computational cost. Our refinement algorithm can further improve the state of the art method's accuracy both on Cityscapes and PASCAL VOC datasets. In evaluating some small objects, our method surpasses most of the state of the art methods.