Texture-Enhanced Light Field Super-Resolution With Spatio-Angular Decomposition Kernels

Abstract

Despite the recent progress in light field super-resolution (LFSR) achieved by convolutional neural networks, the correlation information of LF images has not been sufficiently studied and exploited due to the complexity of 4-D LF data. To cope with such high-dimensional LF data, most of the existing LFSR methods resorted to decomposing it into lower dimensions and subsequently performing optimization on the decomposed subspaces. However, these methods are inherently limited as they neglected the characteristics of the decomposition operations and only utilized a limited set of LF subspaces ending up failing to sufficiently extract spatio-angular features and leading to a performance bottleneck. To overcome these limitations, in this article, we comprehensively discover the potentials of LF decomposition and propose a novel concept of decomposition kernels. In particular, we systematically unify the decomposition operations of various subspaces into a series of such decomposition kernels, which are incorporated into our proposed decomposition kernel network (DKNet) for comprehensive spatio-angular feature extraction. The proposed DKNet is experimentally verified to achieve considerable improvements compared with state-of-the-art methods. To further improve DKNet in producing more visually pleasing LFSR results, based on the VGG network, we propose a light field VGG (LFVGG) loss to guide the texture-enhanced DKNet (TE-DKNet) to generate rich authentic textures and enhance LF images’ visual quality significantly. We also propose an indirect evaluation metric by taking advantage of LF material recognition to objectively assess the perceptual enhancement brought by the LFVGG loss.