Graph-based modelling of superpixels for automatic identification of empty shelves in supermarkets

Abstract

Automatic detection of empty spaces (gaps) between the displayed products as seen in the images of shelves of a supermarket is an interesting image segmentation problem. This paper presents the first known attempt to solve this commercially relevant challenge. The shelf image is first over-segmented into a number of superpixels to construct a graph of superpixels (SG). Subsequently, a graph convolutional network and a Siamese network are built to process the SG. Finally, a structural support vector machine based inference model is formulated based on SG for segmenting the gap and non-gap regions. In order to validate our method, we manually annotate the images of shelves of three benchmark datasets of retail products. We have achieved ∼70 to ∼85% segmentation accuracy (in terms of mean intersection-over-union) on the annotated datasets. A part of the annotated data is released at https://github.com/gapDetection/gapDetectionDatasets.