Light Field Views Research Articles

Learning zero-shot dense light field reconstruction from heterogeneous imaging

Due to the limited sensor size, light field (LF) imaging usually suffers from an essential trade-off between its spatial and angular resolutions. Recently, some approaches have been proposed to use hybrid lens imaging to enhance spatial resolution, but LF angular reconstruction has not been involved. In this paper, we propose a zero-shot learning-based method for reconstructing a dense LF image from a heterogeneous imaging system that integrates a 2D digital camera and a commercial LF camera. Our main insight is that the high-resolution information of the 2D digital image can assist in enhancing the spatial resolution of the LF image, and further, the spatially enhanced LF image can guide the angular reconstruction of the 2D digital image. To avoid ambiguity, it is pointed out that this paper focuses on the latter, i.e., enhancing the angular resolution of the 2D image. Specifically, we first construct two unsupervised modules to learn the geometric property of the acquired LF image and the correlation between the LF image and 2D image, so as to perform the initial reconstruction. Afterwards, an LF detail refiner is constructed to improve the reconstruction quality, and a cycle consistency constraint is designed to realize unsupervised training. Consequently, we can learn the dense LF reconstruction from the heterogeneous imaging data itself without extra training samples. Experimental results show that the proposed method can faithfully reconstruct high-quality densely-sampled LF images, and surpasses some state-of-the-art methods in both quantitative and qualitative evaluations. Furthermore, the proposed method can be combined with existing LF view stitching methods to further enhance the field of view of the reconstructed LF image. Our code will be released at https://github.com/YeyaoChen/ZLFSR.

Light field super-resolution using complementary-view feature attention

Light field (LF) cameras record multiple perspectives by a sparse sampling of real scenes, and these perspectives provide complementary information. This information is beneficial to LF super-resolution (LFSR). Compared with traditional single-image super-resolution, LF can exploit parallax structure and perspective correlation among different LF views. Furthermore, the performance of existing methods are limited as they fail to deeply explore the complementary information across LF views. In this paper, we propose a novel network, called the light field complementary-view feature attention network (LF-CFANet), to improve LFSR by dynamically learning the complementary information in LF views. Specifically, we design a residual complementary-view spatial and channel attention module (RCSCAM) to effectively interact with complementary information between complementary views. Moreover, RCSCAM captures the relationships between different channels, and it is able to generate informative features for reconstructing LF images while ignoring redundant information. Then, a maximum-difference information supplementary branch (MDISB) is used to supplement information from the maximum-difference angular positions based on the geometric structure of LF images. This branch also can guide the process of reconstruction. Experimental results on both synthetic and real-world datasets demonstrate the superiority of our method. The proposed LF-CFANet has a more advanced reconstruction performance that displays faithful details with higher SR accuracy than state-of-the-art methods.

Light Field View Synthesis Using the Focal Stack and All-in-Focus Image.

Light field reconstruction and synthesis algorithms are essential for improving the lower spatial resolution for hand-held plenoptic cameras. Previous light field synthesis algorithms produce blurred regions around depth discontinuities, especially for stereo-based algorithms, where no information is available to fill the occluded areas in the light field image. In this paper, we propose a light field synthesis algorithm that uses the focal stack images and the all-in-focus image to synthesize a 9 × 9 sub-aperture view light field image. Our approach uses depth from defocus to estimate a depth map. Then, we use the depth map and the all-in-focus image to synthesize the sub-aperture views, and their corresponding depth maps by mimicking the apparent shifting of the central image according to the depth values. We handle the occluded regions in the synthesized sub-aperture views by filling them with the information recovered from the focal stack images. We also show that, if the depth levels in the image are known, we can synthesize a high-accuracy light field image with just five focal stack images. The accuracy of our approach is compared with three state-of-the-art algorithms: one non-learning and two CNN-based approaches, and the results show that our algorithm outperforms all three in terms of PSNR and SSIM metrics.

Enabling Five-Degree-of-Freedom From Low-Dimensional Light Fields With Extended Light Ray Acquisition Field-of-View

Conventional captured-image-based virtual reality (VR) systems hardly support free movement of the viewpoint, greatly reducing the user’s sense of reality. A light field (LF), which generates a novel view at a free viewpoint through a combination of light rays, is a suitable approach for increasing the degree of freedom (DoF) in captured-image-based VR. Previous LF-based VR systems have increased LF dimensions or used various acquisition layouts to obtain as many light rays as possible and thereby expand the renderable view range. However, these attempts are costly and time-consuming. This paper enables 5-DoF outcomes from a low-dimensional LF by increasing the field-of-view during light ray acquisition. The method starts with the simple idea that if a three-dimensional (3D) LF is constructed with 360° images that can capture all directions, the renderable range can be widened in a cost-efficient manner. However, using light rays with a large incident angle in a 3D LF can cause visual artifacts. To cope with this problem, this paper initially defines and analyzes the visual artifacts. While these errors were simply ignored in the past, they become particularly noticeable when using light rays with large incident angles. Thus, a careful analysis of the errors and their impacts is required. Additionally, this paper proposes two schemes to deal with visual artifacts based on the error analysis: a horizontal LF and vertical depth (HLF-VD) view generation scheme that complements the structural limitation of 3D LFs and can address all types of errors, and a hybrid 3D LF connection scheme that effectively reduces connection errors by controlling the connection position of neighboring 3D LFs. The proposed system is implemented and verified in various real-world scenarios, and the experimental results show that the user’s movement is naturally reflected at a level similar to that of high-dimensional LF, with minimal visual artifacts.

Learning a Compact Spatial-Angular Representation for Light Field

The recent emergence of light field technology has led to new opportunities for immersive visual communication that has a need for high spatial and angular resolution, both of which contribute to a large image storage footprint and high-latency transmission. Task-driven downsampling methods have been proposed as a solution, and have shown improvements in single-image restoration. However, they are inevitable to disregard light field's intrinsic properties in the corresponding tasks. In this paper, we propose a light-field-specific task-driven downsampling framework, called the LFCrNet. The LFCrNet operates on a learning-based decreasing and increasing resolution in an end-to-end manner in order to utilize a cross-view asymmetric sampling technique. In detail, it separates raw data into disparity and non-disparity patterns by measuring pixel-wise residuals between the sub-aperture central view and auxiliary views. Then, a chain of three-dimensional (3-D) deformable residual blocks (DRBs) is used to extract disparity features and manage these features regard of their intrinsic property individually. Afterwards, they are compacted into spatio-angular domains through a 3-D deformable downsampler (3-DDS). The non-disparity information is integrated into a separate pipeline that leverages spatial similarity across multiple light field views. This technique is capable of preserving specific occlusion components, and subsequently, restoring them using a learning-based upscaling method to generate a high-quality reconstruction. In general, our method has shown superior performance on multiple open-source datasets by a significant margin.

Intra-Inter View Interaction Network for Light Field Image Super-Resolution

Light field (LF) cameras, which can record real-word scenes from multiple viewpoints in a single shot, are widely used in 3D reconstruction, re-focusing, and virtual reality <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">etc</i> . However, the inherent trade-off between spatial resolution and angular resolution of LF images hinders their applications for scenarios requiring high resolutions. In this paper, we propose a novel intra-inter view interaction network for LF image super-resolution, termed as LF-IINet, to exploit the correlations among all views and simultaneously preserve the parallax structure of LF views. The proposed LF-IINet consists of two parallel branches. Specifically, the top branch extracts global inter-view information, and the bottom branch first independently maps each view to deep representations and then models the correlations among all intra-view features via proposed multi-view context block (MCB). The two branches interact with each other by proposed inter-assist-intra feature updating module (IntraFUM, where the intra feature are updated with the assistance of the inter feature) and intra-assist-inter feature updating module (InterFUM, where the inter feature are updated with the assistance of the intra feature). In this way, our LF-IINet incorporates rich angular and spatial information for LF image super-resolution. Extensive comparison with state-of-the-art methods demonstrates that our method achieves superior performance visually and quantitatively. Furthermore, quantitative results also show that our method is effective for LF images with either small or large disparities. Our code is shared in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/GaoshengLiu/LF-IINet</uri> .

Geometry-aware view reconstruction network for light field image compression

Light Field (LF) imaging empowers many attractive applications by simultaneously recording spatial and angular information of light rays. In order to meet the challenges of LF storage and transmission, many view reconstruction-based LF compression methods are put forward. However, occlusion issue and under-exploitation of LF rich structure information limit the view reconstruction qualities, which further influence LF compression efficiency. In order to alleviate these problems, in this paper, we propose a geometry-aware view reconstruction network for LF compression. In our method, only sparsely-sampled LF views are encoded, which are further used as priors to reconstruct the un-sampled LF views at the decoder side. The proposed reconstruction process contains two stages including geometry-aware reconstruction and texture refinement. The geometry-aware reconstruction stage utilizes a multi-stream framework, which can fully explore LF spatial-angular, location and geometry information. The texture refinement stage can adequately fuse such rich LF information to further improve LF reconstruction quality. Comprehensive experimental results validate the superiority of the proposed method. The rate-distortion performance and the perceptual quality of reconstructed views further demonstrate that the proposed method can save more bitrate while increasing LF reconstruction quality.

Light field image super-resolution based on raw data with transformers.

Light field (LF) image super-resolution (SR) can improve the limited spatial resolution of LF images by using complementary information from different perspectives. However, current LF image SR methods only use the RGB data to implicitly exploit the information among different perspectives, without paying attention to the information loss from raw data to RGB data and the explicit structure information utilization. To address the first issue, a data generation pipeline is developed to collect LF raw data for LF image SR. In addition, to make full use of the multiview information, an end-to-end convolutional neural network architecture (namely, LF-RawSR) is proposed for LF image SR. Specifically, an aggregated module is first used to fuse the angular information based on a volume transformer with plane sweep volume. Then the aggregated feature is warped to all LF views using a cross-view transformer for nonlocal dependencies utilization. The experimental results demonstrate that our method outperforms existing state-of-the-art methods with a comparative computational cost, and fine details and clear structures can be restored.

Light-field spatial super-resolution via enhanced spatial-angular separable convolutional network

Light-field images captured by light-field cameras usually suffer from low spatial resolution due to the inherent limited sensor resolution. Light-field spatial super-resolution thus becomes increasingly desirable for subsequent applications. Although continuous progress has been achieved, the existing methods still failed to thoroughly explore the coherence among light-field views. To address this issue, we propose an efficient neural network for light-field spatial super-resolution, in which the spatial and angular information can be fully exploited by repeatedly alternating spatial and angular domain. Specifically, an enhanced spatial-angular separable convolution block is proposed to efficiently exploit the correlation information between sub-aperture images. Moreover, a multi-scale feature extraction block is introduced to extract feature representations at different scales and capture rich texture and semantic information. Experimental results on both synthetic and real-world light-field datasets demonstrate that the proposed method outperforms other state-of-the-art methods with higher peak signal-to-noise ratio (PSNR)/structural similarity (SSIM) values and fewer parameters.

Light field occlusion removal network via foreground location and background recovery

Occlusion removal in an image can aid in facilitating the robustness of numerous computer vision tasks, e.g., detection and tracking in surveillance. However, the invisible property of contents behind the occlusions limits occlusion removal from the single view. Recently, the emerging light field (LF) data, which contains rich multi-view perception of the scene, provides potential solution for this challenge. To better exploit the capability of occlusion location and occluded contents recovery from LF data, in this paper, we propose a LF occlusion removal network (LFORNet), which consists of three key sub-networks: the foreground occlusion location (FOL) sub-network, the background content recovery (BCR) sub-network, and the refinement sub-network. Specifically, both FOL sub-network and BCR sub-network explore the multi-view information of LF data, and thus they are constructed with the same network structure to estimate the occlusion mask and the coarse occluded contents map, respectively. The refinement sub-network aggregates the above two outputs to obtain the refined occlusion removal. Meanwhile, we use multi-angle view stacks as the input of the network, which can make full use of the inherent information among the LF views. Experimental results show that our method is suitable for different sizes of occlusions, and surpasses the state-of-the-art approaches in both synthetic and real-world scenes.

A novel hierarchical light field coding scheme based on hybrid stacked multiplicative layers and Fourier disparity layers for glasses-free 3D displays

We present a novel hierarchical coding scheme for light fields based on transmittance patterns of low-rank multiplicative layers and Fourier disparity layers. The proposed scheme identifies multiplicative layers of light field view subsets optimized using convolutional neural networks for different scanning orders. Our approach exploits the hidden low-rank structure in the multiplicative layers obtained from the subsets of different scanning patterns. The spatial redundancies in the multiplicative layers can be efficiently removed by performing low-rank approximation at different ranks on the Krylov subspace. The intra-view and inter-view redundancies between approximated layers are further removed by HEVC encoding. Next, a Fourier disparity layer representation is constructed from the first subset of the approximated light field based on the chosen hierarchical order. Subsequent view subsets are synthesized by modeling the Fourier disparity layers that iteratively refine the representation with improved accuracy. The critical advantage of the proposed hybrid layered representation and coding scheme is that it utilizes not just spatial and temporal redundancies in light fields, but also efficiently exploits intrinsic similarities among neighboring sub-aperture images in both horizontal and vertical directions as specified by different predication orders. In addition, the scheme is flexible to realize a range of multiple bitrates at the decoder within a single integrated system. Comparison with state-of-the-art light field coders exhibits superior compression performance of the proposed scheme for real-world light fields. We achieve substantial bitrate savings and also maintain good light field reconstruction quality.

Progressive Multi-Scale Fusion Network for Light Field Super-Resolution

Light field (LF) cameras can record multi-view images from a single scene, and these images can provide spatial and angular information to improve the performance of image super-resolution (SR). However, it is a challenge to incorporate distinctive information from different LF views. At the same time, due to the inherent resolution of the image sensor, high spatial and angular resolution are trade-off problems. In this paper, we propose a progressive multi-scale fusion network (PMFN) to improve the LFSR performance. Specifically, a progressive feature fusion block (PFFB) based on an encoder-and-decoder structure is designed to implicitly align disparities and integrate complementary information between complementary views. The core module of the PFFB is a dual-branch multi-scale fusion module (DMFM), which can integrate the information from a reference view and auxiliary views to produce a fusion feature. Each DMFM consists of two parallel branches, which have different receptive fields to fuse hierarchical features from complementary views. Three DMFMs with a dense connection are used in the PFFB, which can fully exploit multi-level features to improve the SR performance. Experimental results on both synthetic and real-world datasets demonstrate that the proposed model achieves state-of-the-art performance among existing methods. Moreover, quantitative results show that our method can also generate faithful details.

A light field three-dimensional ranging system based on regularization

A high-precision light field three-dimensional ranging system based on regularization is proposed. This method transforms the sub-aperture image resolution problem into a global optimization problem. The objective function is constructed to capture the relationships between the light field views. The complementary information in the light field views is then utilized to improve the spatial resolution of the light field. In particular, regularizations of fuzzy matrix-based and optical flow-field-based use of information from multiple viewpoints can effectively enhance the light field structure of the system. The experimental results show that the proposed method can perform scene distance measurement accuracy and has better depth estimation. The proposed method effectively solves image depth distortion in traditional light field imaging. It eliminates the influence of low resolution of sub-aperture images on depth estimation to the maximum extent.

Unsupervised disparity estimation from light field using plug-and-play weighted warping loss

We investigated disparity estimation from a light field using a convolutional neural network (CNN). Most of the methods implemented a supervised learning framework, where the predicted disparity map was compared directly to the corresponding ground-truth disparity map in the training stage. However, light field data accompanied with ground-truth disparity maps were insufficient and rarely available for real-world scenes. The lack of training data resulted in limited generality of the methods trained with them. To tackle this problem, we took a simple Figure-and-play approach to remake a supervised method into an unsupervised (self-supervised) one. We replaced the loss function of the original method with one that does not depend on the ground-truth disparity maps. More specifically, our loss function is designed to indirectly evaluate the accuracy of the disparity map by using warping errors among the input light field views. We designed pixel-wise weights to properly evaluate the warping errors in the presence of occlusions, and an edge loss to encourage edge alignment between the image and the disparity map. As a result of this unsupervised learning framework, our method can use more abundant training datasets (even those without ground-truth disparity maps) than the original supervised method. Our method was evaluated on computer-generated scenes (4D Light Field Benchmark) and real-world scenes captured by Lytro Illum cameras. Our method achieved the state-of-the-art performance as an unsupervised method on the benchmark. We also demonstrated that our method can estimate disparity maps more accurately than the original supervised method for various real-world scenes.

Occlusion-Aware Unsupervised Learning of Depth From 4-D Light Fields.

Depth estimation is a fundamental issue in 4-D light field processing and analysis. Although recent supervised learning-based light field depth estimation methods have significantly improved the accuracy and efficiency of traditional optimization-based ones, these methods rely on the training over light field data with ground-truth depth maps which are challenging to obtain or even unavailable for real-world light field data. Besides, due to the inevitable gap (or domain difference) between real-world and synthetic data, they may suffer from serious performance degradation when generalizing the models trained with synthetic data to real-world data. By contrast, we propose an unsupervised learning-based method, which does not require ground-truth depth as supervision during training. Specifically, based on the basic knowledge of the unique geometry structure of light field data, we present an occlusion-aware strategy to improve the accuracy on occlusion areas, in which we explore the angular coherence among subsets of the light field views to estimate initial depth maps, and utilize a constrained unsupervised loss to learn their corresponding reliability for final depth prediction. Additionally, we adopt a multi-scale network with a weighted smoothness loss to handle the textureless areas. Experimental results on synthetic data show that our method can significantly shrink the performance gap between the previous unsupervised method and supervised ones, and produce depth maps with comparable accuracy to traditional methods with obviously reduced computational cost. Moreover, experiments on real-world datasets show that our method can avoid the domain shift problem presented in supervised methods, demonstrating the great potential of our method. The code will be publicly available at https://github.com/jingjin25/LFDE-OccUnNet.

Deep Residual Architecture Using Pixel and Feature Cues for View Synthesis and Temporal Interpolation

In this paper, we propose a deep residual architecture that can be used both for synthesizing high quality angular views in light fields and temporal frames in classical videos. The proposed framework consists of an optical flow estimator optimized for view synthesis, a trainable feature extractor and a residual convolutional network for pixel and feature-based view reconstruction. Among these modules, the fine-tuning of the optical flow estimator specifically for the view synthesis task yields scene depth or motion information that is well optimized for the targeted problem. In cooperation with the end-to-end trainable encoder, the synthesis block employs both pixel-based and feature-based synthesis with residual connection blocks, and the two synthesized views are fused with the help of a learned soft mask to obtain the final reconstructed view. Experimental results with various datasets show that our method performs favorably against other state-of-the-art (SOTA) methods with a large gain for light field view synthesis. Furthermore, with a little modification, our method can also be used for video frame interpolation, generating high quality frames compared with SOTA interpolation methods.

Light Field View Synthesis Using the Focal Stack and All-in-Focus Image

Light Field View Synthesis Using the Focal Stack and All-in-Focus Image

Attention Mechanism-Based Light-Field View Synthesis

The angular information of light lost in conventional images but preserved and stored in light-fields plays an instrumental role in many applications such as depth estimation, 3D reconstruction and post-capture refocusing. However, the limited angular resolution of light-field images due to the consumer hardware limitations is a major drawback in its widespread adoption. In this article, we present a novel deep learning-based light-field view synthesis method from a sparse set of input views. Our proposed method, an end-to-end trainable network, utilizes convolutional block attention modules to enhance the built-in depth image-based rendering. The proposed convolutional block attention module consists of two attention modules sequentially applied in the channel and spatial dimensions to focus the network on critical features. The proposed network architecture is a combination of three sub-networks, one for stereo feature extraction, another for disparity estimation, and the last for attention-based refinement. We present two schemes for the refinement network that perform equally well but differ in the number of parameters by 1.2% and execution time by 44%. Quantitative and qualitative results on four challenging real-world datasets show the superiority of the proposed method. Our proposed method shows considerable PSNR gains, around 1 dB compared to the state of the art and around 0.5 dB over our previous method LFVS-AM. In addition, ablation studies demonstrate the effectiveness of each module of the proposed method. Finally, the parallax edge precision-recall curve shows that our method better preserves parallax details.

A Fast View Synthesis Implementation Method for Light Field Applications

View synthesis (VS) for light field images is a very time-consuming task due to the great quantity of involved pixels and intensive computations, which may prevent it from the practical three-dimensional real-time systems. In this article, we propose an acceleration approach for deep learning-based light field view synthesis, which can significantly reduce calculations by using compact-resolution (CR) representation and super-resolution (SR) techniques, as well as light-weight neural networks. The proposed architecture has three cascaded neural networks, including a CR network to generate the compact representation for original input views, a VS network to synthesize new views from down-scaled compact views, and a SR network to reconstruct high-quality views with full resolution. All these networks are jointly trained with the integrated losses of CR, VS, and SR networks. Moreover, due to the redundancy of deep neural networks, we use the efficient light-weight strategy to prune filters for simplification and inference acceleration. Experimental results demonstrate that the proposed method can greatly reduce the processing time and become much more computationally efficient with competitive image quality.

Learning from EPI-Volume-Stack for Light Field image angular super-resolution

Light Field (LF) image angular super-resolution aims to synthesize a high angular resolution LF image from a low angular resolution one, and is drawing increased attention because of its wide applications. In order to reconstruct a high angular resolution LF image, many learning based LF image angular super-resolution methods have been proposed. However, most existing methods are based on LF Epipolar Plane Image or Epipolar Plane Image volume representation, which underuse the LF image structure. The LF view spatial correlation and neighboring LF views angular correlations which can reflect LF image structure are not fully explored, which reduces LF angular super-resolution quality. In order to alleviate this problem, this paper introduces an Epipolar Plane Image Volume Stack (EPI-VS) representation for LF angular super-resolution. The EPI-VS is constituted by arranging all LF views in a raster order, which benefits in exploring LF view spatial correlation and neighboring LF views angular correlations. Based on such representation, we further propose an LF angular super-resolution network. 3D convolutions are applied in the whole super-resolution network to better accommodate the input EPI-VS data and allow information propagation between two spatial and one directional dimensions of EPI-VS data. Extensive experiments on synthetic and real-world LF scenes demonstrate the effectiveness of the proposed network. Moreover, we also illustrate the superiority of our network by applying it in scene depth estimation task.