Real-time panoptic segmentation with relationship between adjacent pixels and boundary prediction

Abstract

Panoptic segmentation has recently received increasing attention since it generates coherent scene segmentation by unifying semantic and instance segmentation. The most popular methods for panoptic segmentation are currently based on an instance segmentation framework with a semantic segmentation branch in parallel. However, these methods are too bloated for real-world applications. In this paper, we propose a simple yet effective fully convolutional network for fast panoptic segmentation. Instead of directly generating the mask for each instance, we leverage a simple graph convolutional layer to construct a pixel relationship head to predict the relationship between two adjacent pixels and determine whether they belong to the same instance. Besides, we leverage boundary information to enhance supervision information and help our method distinguish adjacent objects. Combining predicted category labels for each pixel from the semantic segmentation branch, we can generate a unified panoptic segmentation mask in a parameter-free step. We demonstrate our method’s effectiveness on MS COCO dataset and Cityscapes dataset, which obtain competitive results.