Adaptive pixel aggregation for joint spatial and angular super-resolution of light field images

Abstract

Recent advancements in Light Field (LF) super-resolution (SR) have significantly enhanced the spatial size and angular sampling rate in LF imagery. Typically, most methods treat spatial and angular SR as separate tasks. However, the two tasks are closely related and can be jointly optimized by leveraging the intrinsic high-dimensional property of LF images. Existing methods for joint spatial and angular SR of LF images also suffer from the limitations of disparity-based warping operations. To address these challenges, we propose a novel one-stage approach for directly synthesizing densely-sampled high-resolution LF images from sparsely-sampled low-resolution observations. Specifically, we propose to adaptively aggregate advantageous pixels from different sub-aperture images of input LF for each novel view interpolation. This adaptive strategy enables our method to effectively incorporate the spatial and angular correlations of input views. Additionally, to enhance the parallax structure of reconstructed LF images, we propose a feature separation and interaction module to refine the intermediate features. Experimental results on public datasets demonstrate that our proposed method can achieve state-of-the-art performance both numerically and visually compared with previous methods. Our codes are available at https://github.com/GaoshengLiu/LFSR.