ALFO: Adaptive Light Field Over-Segmentation

Maryam Hamad,Paulo Nunes,Luis Ducla Soares,Caroline Conti

doi:10.1109/access.2021.3114324

Abstract

Automatic image over-segmentation into superpixels has attracted increasing attention from researchers to apply it as a pre-processing step for several computer vision applications. In 4D Light Field (LF) imaging, image over-segmentation aims at achieving not only superpixel compactness and accuracy but also cross-view consistency. Due to the high dimensionality of 4D LF images, depth information can be estimated and exploited during the over-segmentation along with spatial and visual appearance features. However, balancing between several hybrid features to generate robust superpixels for different 4D LF images is challenging and not adequately solved in existing solutions. In this paper, an automatic, adaptive, and view-consistent LF over-segmentation method based on normalized LF cues and $K$ -means clustering is proposed. Initially, disparity maps for all LF views are estimated entirely to improve superpixel accuracy and consistency. Afterwards, by using $K$ -means clustering, a 4D LF image is iteratively divided into regular superpixels that adhere to object boundaries and ensure cross-view consistency. Our proposed method can automatically adjust the clustering weights of the various features that characterize each superpixel based on the image content. Quantitative and qualitative results on several 4D LF datasets demonstrate outperforming performance of the proposed method in terms of superpixel accuracy, shape regularity and view consistency when using adaptive clustering weights, compared to the state-of-the-art 4D LF over-segmentation methods.

Highlights

Image segmentation is a process of dividing the scene into several coherent regions according to some criteria
In real Light Field (LF) images, where the disparity maps are affected by noise or non-even lighting across views, Adaptive LF Over-segmentation (ALFO) may generate an imprecise segmentation and superpixels may not adhere well to the boundaries when there are disparity ambiguities
K-means clustering in local searching can be parallelized, as shown in [7] for the proposed 2D superpixel method and in [8] for 4D LF images; it is expected that our method can be further optimized to generate faster over-segmentation and reduce the overall subsequent editing complexity

Summary

Introduction

Image segmentation is a process of dividing the scene into several coherent regions according to some criteria. Available image segmentation algorithms in the literature require different levels of supervision to suit different types of applications These algorithms can be classified into supervised [5], semisupervised [6], and unsupervised (automatic) [7], [8], based on the need for pre-trained labels or human interactions. Preserving all image boundaries during the over-segmentation could be challenging, since the used ground truth labels for training are usually segmented in a more semantically meaningful level (e.g., object level) Their performance is competitive compared to unsupervised methods, the generalization of the network to over-segment different datasets is still a challenge to be further studied

Methods

Results

Discussion

Conclusion