Light field reconstruction using hierarchical features fusion

Abstract

Light field imagery has attracted increasing attention for its capacity of simultaneously capturing intensity values of light rays from multiple directions. Such imagery technique has become widely accessible with the emergence of consumer-grade devices, e.g. Lytro, and the Virtual Reality (VR) / Augmented Reality (AR) areas. Light field reconstruction is a critical topic to mitigate the trade-off problem between the spatial and angular resolutions. Learning-based methods have attained outstanding performance among the recently proposed methods, however, the state-of-the-art methods still suffer from heavy artifacts in the case of occlusion. This is likely to be a consequence of failure in capturing the semantic information from the limited spatial receptive field during training. It is crucial for light field reconstruction to learn semantic features and understand a wider context in both the angular and spatial dimensions. To address this issue, we introduce a novel end-to-end U-Net with SAS network (U-SAS-Net) to extract and fuse hierarchical features, both local and semantic, from a relatively large receptive field while establishing the relation of the correlated sub-aperture images. Experimental results on extensive light field datasets demonstrate that our method produces a state-of-the-art performance that exceeds the previous works by more than 0.6 dB PSNR with the fused hierarchical features, especially the semantic features for handling scenes with occlusion and the local features for recovering the rich details. Meanwhile, our method is at a substantially lower cost which takes 48% parameters and less than 10% computation of the previous state-of-the-art method.