Learning zero-shot dense light field reconstruction from heterogeneous imaging

Abstract

Due to the limited sensor size, light field (LF) imaging usually suffers from an essential trade-off between its spatial and angular resolutions. Recently, some approaches have been proposed to use hybrid lens imaging to enhance spatial resolution, but LF angular reconstruction has not been involved. In this paper, we propose a zero-shot learning-based method for reconstructing a dense LF image from a heterogeneous imaging system that integrates a 2D digital camera and a commercial LF camera. Our main insight is that the high-resolution information of the 2D digital image can assist in enhancing the spatial resolution of the LF image, and further, the spatially enhanced LF image can guide the angular reconstruction of the 2D digital image. To avoid ambiguity, it is pointed out that this paper focuses on the latter, i.e., enhancing the angular resolution of the 2D image. Specifically, we first construct two unsupervised modules to learn the geometric property of the acquired LF image and the correlation between the LF image and 2D image, so as to perform the initial reconstruction. Afterwards, an LF detail refiner is constructed to improve the reconstruction quality, and a cycle consistency constraint is designed to realize unsupervised training. Consequently, we can learn the dense LF reconstruction from the heterogeneous imaging data itself without extra training samples. Experimental results show that the proposed method can faithfully reconstruct high-quality densely-sampled LF images, and surpasses some state-of-the-art methods in both quantitative and qualitative evaluations. Furthermore, the proposed method can be combined with existing LF view stitching methods to further enhance the field of view of the reconstructed LF image. Our code will be released at https://github.com/YeyaoChen/ZLFSR.