Light Field Representation Research Articles

LF2MV: Learning an Editable Meta-View Towards Light Field Representation.

Light fields are 4D scene representations that are typically structured as arrays of views or several directional samples per pixel in a single view. However, this highly correlated structure is not very efficient to transmit and manipulate, especially for editing. To tackle this issue, we propose a novel representation learning framework that can encode the light field into a single meta-view that is both compact and editable. Specifically, the meta-view composes of three visual channels and a complementary meta channel that is embedded with geometric and residual appearance information. The visual channels can be edited using existing 2D image editing tools, before reconstructing the whole edited light field. To facilitate edit propagation against occlusion, we design a special editing-aware decoding network that consistently propagates the visual edits to the whole light field upon reconstruction. Extensive experiments show that our proposed method achieves competitive representation accuracy and meanwhile enables consistent edit propagation.

Neural Subspaces for Light Fields.

We introduce a framework for compactly representing light field content with the novel concept of neural subspaces. While the recently proposed neural light field representation achieves great compression results by encoding a light field into a single neural network, the unified design is not optimized for the composite structures exhibited in light fields. Moreover, encoding every part of the light field into one network is not ideal for applications that require rapid transmission and decoding. We recognize this problem's connection to subspace learning. We present a method that uses several small neural networks, specializing in learning the neural subspace for a particular light field segment. Moreover, we propose an adaptive weight sharing strategy among those small networks, improving parameter efficiency. In effect, this strategy enables a concerted way to track the similarity among nearby neural subspaces by leveraging the layered structure of neural networks. Furthermore, we develop a soft-classification technique to enhance the color prediction accuracy of neural representations. Our experimental results show that our method better reconstructs the light field than previous methods on various light field scenes. We further demonstrate its successful deployment on encoding light fields with irregular viewpoint layout and dynamic scene content.

Multi-scale foreground-background separation for light field depth estimation with deep convolutional networks

There are three viewpoints to divide the depth from light field (DFLF) methods: light field (LF) representation, signal processing, and learning viewpoints. In the signal processing viewpoint in details, the DFLF methods can be divided into three groups: cost-based, foreground-background separation (FBS)-based, and depth model-based methods. Although the deep learning techniques are widely implemented in the cost-based and depth model-based methods for the DFLF, they are not implemented in the FBS-based method, yet. In this paper, we propose a DFLF method based on the FBS with the deep learning techniques. Based on an observation that the angular aliasing artifacts are reduced in the resulting disparity maps obtained from spatially low-scale LF images, multi-scale (MS) FBS score maps are used in the proposed methods. To combine the MS FBS score maps into a single FBS score maps, four directional LF gradients, features from the 4-D LF, and features from the center view images are utilized. Based on the sensitivity and specificity loss, the gradient matching loss between the predicted and ground truth disparity maps is used to improve the performance around the occlusion boundary in the resulting disparity maps. The experimental results show that the proposed method performs reasonably in both the synthetic and real LF datasets. Especially, in the texture-less region in the real dataset, the resulting disparity maps obtained by the proposed method show that the artifacts are reduced compared to those by the existing DFLF methods.

Dense light field reconstruction based on epipolar focus spectrum

Existing light field (LF) representations, such as epipolar plane image (EPI) and sub-aperture images, do not consider the structural characteristics across the views, so they usually require additional disparity and spatial structure cues for follow-up tasks. Besides, they have difficulties dealing with occlusions or large disparity scenes. To this end, this paper proposes a novel Epipolar Focus Spectrum (EFS) representation by rearranging the EPI spectrum. Different from the classical EPI representation where an EPI line corresponds to a specific depth, there is a one-to-one mapping from the EFS line to the view. By exploring the EFS sampling task, the analytical function is derived for constructing a non-aliasing EFS. To demonstrate its effectiveness, we develop a trainable EFS-based pipeline for light field reconstruction, where a dense light field can be reconstructed by compensating the missing EFS lines given a sparse light field, yielding promising results with cross-view consistency, especially in the presence of severe occlusion and large disparity. Experimental results on both synthetic and real-world datasets demonstrate the validity and superiority of the proposed method over SOTA methods.

Learning Light Fields for Improved Lane Detection

Robust lane detection is imperative for the realization of intelligent transportation. Recently, vision-based systems that employ deep convolution neural networks (CNNs) for lane detection have made considerable progress. However, for better generalization under various road conditions learning-based methods require excessive training data, which becomes non-trivial in challenging conditions such as illumination variation, shadows, false lane lines, and worn lane markings, etc. In this paper, we propose a light field (LF) based lane detection method that utilizes the additional angular information for improved prediction and increased robustness. Two different LF representations are investigated to study the possibility of maximum performance improvement and minimal additional computation cost and data labeling efforts. Experimental results successfully demonstrate that the proposed approach improves the prediction of the lane line point coordinates and is significantly robust against the aforementioned adverse conditions.

Light Field Image Super-Resolution Network via Joint Spatial-Angular and Epipolar Information

This paper proposes a novel efficient convolutional neural network (CNN) for light field (LF) spatial super-resolution (SR). Due to the high dimensionality of LF data, it is important to decompose the original 4D LF data structure into a lower data subspace helping to reduce memory and computational costs. In this context, we propose a novel hybrid CNN structure working on different 2D data representations to fully handle the 4D structure. Our proposed network can simultaneously exploit information from three different types of 4D LF representations: spatial, angular, and epipolar plane image (EPI) information. Based on pixel similarities of each data subspace, we propose an inter-intra spatial feature extractor (II-SFE) and an inter-intra angular feature extractor (II-AFE) that can exploit the pixel correlations among the spatial-angular domain. Moreover, based on the correlated epipolar geometry among pixels, we also propose a multi-orientation epipolar feature extractor (MO-EFE) by decomposing an LF image into four orientation stacks of views and feeding them into different network branches to learn rich sub-pixel information in the EPI domain. The extracted features of each 2D subspace are integrated by our proposed attention fusion module (AFM), which adaptively rescales the embedding weights based on their correlations. Experimental results on both small- and wide-baseline benchmark LF datasets confirm the effectiveness of the proposed method compared to the well-known state-of-the-art methods in terms of objective and subjective performance.

Visualization of an Ultra-Short THz Beams with a Radiation Propagation Analysis of the Novel Israeli Free Electron Laser

Tera Hertz radiation is currently the most researched and useful area in almost all fields of science and industry. The additional challenge is expressed in the form of radiation, pulses of femto-seconds in length are supposed to pass through a transmission line (TL) most efficiently, at a wide range of frequencies. These are complex beams, which make up the electromagnetic (EM) field, represented in the frequency domain in terms of cavity eigenmodes. A simulation allows to describe of the phase-amplitude and spectral characteristics of multimode radiation free-electron laser (FEL) operating in various operational parameters. The analysis is performed through the transmission of optical rays accurately, with each ray being characterized by amplitude, position, and angle in 3D space. A light field representation of a complex EM field is obtained via Wigner Distribution Function, which allows to describe of the dynamics of field evolution in future propagation by a ray tracing (RT) method. The final diagnostics will determine the design of the TL to be assembled in an innovative accelerator under construction at the Schlesinger Family Center for Compact Accelerators, Radiation Sources, and Applications.

A Practical Light Field Representation and Coding Scheme with an Emphasis on Refocusing

Light field images have drawn a great deal of interest owing to their flexible post-processing and versatile functionalities. However, the tremendous data volume has imposed a practical limit in applications, which requires a better way to represent and compress the data. In this paper, we propose a practical representation and coding scheme for light field data with a special emphasis on retaining the refocusing functionality as much as possible. Under the proposed method, light field data are represented and compressed in the form of a rendered all-in-focus image and a depth map. Before encoding, the all-in-focus image is rendered accurately based on the depth map using the proposed difference focus measure. At the decoder, a focal stack comprising multiple images having different focus levels can be reconstructed by applying a defocusing function to the compressed all-in-focus image where the defocusing function parameter depends on the selected depth level of the compressed depth map. Compared with well-known state-of-the-art methods that compress the Fourier Disparity Layer or the focal stack, the proposed coding scheme shows much smaller loss in the refocusing capability (16.2% and 17.8% less than two state-of-the-art methods) and provides PSNR improvements of 1.60㏈ to 2.38㏈ at the same compression ratio.

A Hybrid Tucker-VQ Tensor Sketch decomposition model for coding and streaming real world light fields using stack of differently focused images

• Lightfield representation, coding and streaming schemes using a stack of focal images. • Layer pattern optimization from focal stack for efficient streaming on tensor display. • Hybrid Tucker Tensor Sketch Scheme with Vector Quantization for low-rank modeling. • A single integrated, adaptable and flexible system to realize a range of bitrates. Computational multi-view displays involving light fields are a fast emerging choice for 3D presentation of real-world scenes. Tensor autostereoscopic glasses-free displays use just few light attenuating layers in front of a backlight to output high quality light field. We propose three novel schemes, Focal Stack - Hybrid Tucker-TensorSketch Vector Quantization (FS-HTTSVQ), Focal Stack - Tucker-TensorSketch (FS-TTS), and Focal Stack - Tucker Alternating Least-Squares (FS-TALS), for efficient representation, streaming and coding of light fields using a stack of differently focused images. Working with a focal stack instead of the entire light field majorly reduces the data acquisition cost as well as the computation and processing cost. Extensive experiments with real world light field focal stacks demonstrate that proposed novel one-pass Tucker decomposition using TensorSketch with hybrid vector quantization in FS-HTTSVQ, compactly represents the approximated focal stack in codebook form for better transmission and streaming. Encoding with High Efficiency Video Coding (HEVC) eliminates all intrinsic redundancies present in the approximated focal stack. Resultant low-rank approximated and coded focal stack is then employed to analytically optimize layer patterns for the tensor display. The complete end-to-end light field processing pipelines flexibly work for multiple bitrates and are adaptable for a variety of multi-view autostereoscopic platforms. Our schemes exhibit note-worthy performances on focal stacks compared to direct encoding of an entire light field using a standard codec like HEVC.

Underwater large field of view 3D imaging based on fisheye lens

As for the 3D imaging of seabed topography, seabed sunken ships and large underwater buildings, the field of view is a significant technical index of the 3D imaging system. In this paper, a binocular vision imaging system based on fisheye lens is applied to satisfy the requirements for 3D imaging of large underwater scenes. Considering the severe image distortion caused by the large wide-angle of fisheye lens and the failure of pinhole imaging model underwater, the imaging process of underwater fisheye camera is re-described through the parametric representation of four-dimensional light field, and the multi-layer refraction imaging model of underwater binocular stereo vision is established based on fisheye camera. A conversion imaging model is constructed to convert the corrected underwater fisheye image into the equivalent image taken in the air. Combined with the existing stereo matching methods, the converted images realize the 3D reconstruction of a large-scale underwater scene. Experimental results show that the corrected underwater fisheye images have a small average reprojection error, which evidences the effectiveness of the proposed calibration method. In the experiment of 3D reconstruction for the large scene composed of multiple underwater ceramic discs, the reconstructed targets have good 3D imaging shapes, which indicates the developed underwater binocular fisheye camera imaging system has the ability to perform 3D imaging and make observations of large underwater scenes.

Weakly-Supervised Salient Object Detection on Light Fields.

Most existing salient object detection (SOD) methods are designed for RGB images and do not take advantage of the abundant information provided by light fields. Hence, they may fail to detect salient objects of complex structures and delineate their boundaries. Although some methods have explored multi-view information of light field images for saliency detection, they require tedious pixel-level manual annotations of ground truths. In this paper, we propose a novel weakly-supervised learning framework for salient object detection on light field images based on bounding box annotations. Our method has two major novelties. First, given an input light field image and a bounding-box annotation indicating the salient object, we propose a ground truth label hallucination method to generate a pixel-level pseudo saliency map, to avoid heavy cost of pixel-level annotations. This method generates high quality pseudo ground truth saliency maps to help supervise the training, by exploiting information obtained from the light field (including depths and RGB images). Second, to exploit the multi-view nature of the light field data in learning, we propose a fusion attention module to calibrate the spatial and channel-wise light field representations. It learns to focus on informative features and suppress redundant information from the multi-view inputs. Based on these two novelties, we are able to train a new salient object detector with two branches in a weakly-supervised manner. While the RGB branch focuses on modeling the color contrast in the all-in-focus image for locating the salient objects, the Focal branch exploits the depth and the background spatial redundancy of focal slices for eliminating background distractions. Extensive experiments show that our method outperforms existing weakly-supervised methods and most fully supervised methods.

A Flexible Coding Scheme Based on Block Krylov Subspace Approximation for Light Field Displays with Stacked Multiplicative Layers.

To create a realistic 3D perception on glasses-free displays, it is critical to support continuous motion parallax, greater depths of field, and wider fields of view. A new type of Layered or Tensor light field 3D display has attracted greater attention these days. Using only a few light-attenuating pixelized layers (e.g., LCD panels), it supports many views from different viewing directions that can be displayed simultaneously with a high resolution. This paper presents a novel flexible scheme for efficient layer-based representation and lossy compression of light fields on layered displays. The proposed scheme learns stacked multiplicative layers optimized using a convolutional neural network (CNN). The intrinsic redundancy in light field data is efficiently removed by analyzing the hidden low-rank structure of multiplicative layers on a Krylov subspace. Factorization derived from Block Krylov singular value decomposition (BK-SVD) exploits the spatial correlation in layer patterns for multiplicative layers with varying low ranks. Further, encoding with HEVC eliminates inter-frame and intra-frame redundancies in the low-rank approximated representation of layers and improves the compression efficiency. The scheme is flexible to realize multiple bitrates at the decoder by adjusting the ranks of BK-SVD representation and HEVC quantization. Thus, it would complement the generality and flexibility of a data-driven CNN-based method for coding with multiple bitrates within a single training framework for practical display applications. Extensive experiments demonstrate that the proposed coding scheme achieves substantial bitrate savings compared with pseudo-sequence-based light field compression approaches and state-of-the-art JPEG and HEVC coders.

Model-independent light field reconstruction using a generic camera calibration

Abstract Sophisticated and highly specialized optical measuring devices are becoming increasingly important for high-precision manufacturing and environment perception. In particular, light field cameras are experiencing an ever-increasing interest in research and industry as they enable a variety of new measurement methods. Unfortunately, due to their complex structure, their calibration is very difficult and usually precisely tailored to the particular type of light field camera. To overcome these difficulties, we present a method that decodes a light field from the raw data of any light field imaging system without knowing and modeling the internal optical elements. We calibrate the camera using a precise generic calibration method and transform the obtained ray set into an equivalent light field representation. Finally, we reconstruct a rectified light field from the irregularly sampled data and in addition we derive the geometric ray properties as intrinsic camera parameters. Experimental results validate the method by showing that both the information of the observed scene and the geometric structure of the light field are preserved by an adequate rectification and calibration.

General design method for three-dimensional displays with horizontally asymmetric pixel structures

Various 3D displays have been proposed to show realistic and vivid 3D images. Moreover, 3D displays have been applied in various fields including medicine, entertainment, and advertising. Depending on the application, 3D displays have different pixel structures and sizes. In this paper, we present a 3D-display design method that can be applied regardless of the pixel structure and display sizes. The area of the designable 3D display is suggested by the improved 3D image quality. The manufactured displays are used to verify the proposed method. Furthermore, a light field simulation is performed to confirm the area that was not proven by the manufactured displays. With the proposed 3D image-quality model and 3D image simulation by the light field representation, a general design of 3D displays with various pixel structures can be developed.

Compressive HDR Light Field Imaging Using a Single Multi-ISO Sensor

In this paper, we propose a new design for single sensor compressive HDR light field cameras, combining multi-ISO photography with coded mask acquisition, placed in a compressive sensing framework. The proposed camera model is based on a main lens, a multi-ISO sensor and a coded mask located in the optical path between the main lens and the sensor that projects the coded spatio-angular information of the light field onto the 2D sensor. The model encompasses different acquisition scenarios with different ISO patterns and gains. Moreover, we assume that the sensor has a built-in color filter array (CFA), making our design more suitable for consumer-level cameras. We propose a reconstruction algorithm to jointly perform color demosaicing, light field angular information recovery, HDR reconstruction, and denoising from the multi-ISO measurements formed on the sensor. This is achieved by enabling the sparse representation of HDR light fields using an overcomplete HDR dictionary. We also provide two HDR light field data sets: one synthetic data set created using the Blender rendering software with two baselines, and a real light field data set created from the fusion of multi-exposure low dynamic range (LDR) images captured using a Lytro Illum light field camera. Experimental results show that, with a sampling rate as low as 2.67%, using two shots, our proposed method yields a higher light field reconstruction quality compared to the fusion of multiple LDR light fields captured with different exposures, and with the fusion of multiple LDR light fields captured with different ISO settings.

Overview of 4D Light Field Representation

Overview of 4D Light Field Representation

CapsField: Light Field-Based Face and Expression Recognition in the Wild Using Capsule Routing.

Light field (LF) cameras provide rich spatio-angular visual representations by sensing the visual scene from multiple perspectives and have recently emerged as a promising technology to boost the performance of human-machine systems such as biometrics and affective computing. Despite the significant success of LF representation for constrained facial image analysis, this technology has never been used for face and expression recognition in the wild. In this context, this paper proposes a new deep face and expression recognition solution, called CapsField, based on a convolutional neural network and an additional capsule network that utilizes dynamic routing to learn hierarchical relations between capsules. CapsField extracts the spatial features from facial images and learns the angular part-whole relations for a selected set of 2D sub-aperture images rendered from each LF image. To analyze the performance of the proposed solution in the wild, the first in the wild LF face dataset, along with a new complementary constrained face dataset captured from the same subjects recorded earlier have been captured and are made available. A subset of the in the wild dataset contains facial images with different expressions, annotated for usage in the context of face expression recognition tests. An extensive performance assessment study using the new datasets has been conducted for the proposed and relevant prior solutions, showing that the CapsField proposed solution achieves superior performance for both face and expression recognition tasks when compared to the state-of-the-art.

Robust Light Field Watermarking by 4D Wavelet Transform

Unlike common 2D images, the light field representation of a scene delivers spatial and angular description which is of paramount importance for 3D reconstruction. Despite the numerous methods proposed for 2D image watermarking, such methods do not address the angular information of the light field. Hence the exploitation of such methods may cause severe destruction of the angular information. In this paper, we propose a novel method for light field watermarking with extensive consideration of the spatial and angular information. Considering the 4D innate of the light field, the proposed method incorporates 4D wavelet for the purpose of watermarking and converts the heavily-correlated channels from RGB domain to YUV. The robustness of the proposed method has been evaluated against common image processing attacks.

A hybrid learning of multimodal cues for light field depth estimation

Learning based light field depth estimation methods have been proven to be successful, however, most of them focus on the depth cues extraction from a single representation of light field. In our opinion, existing methods are difficult to simultaneously learn various cues with different modalities, which are implied in different representations of light field. In this paper, we formulate the light field depth estimation as a pixel-wise classification task, and introduce a hybrid learning architecture to combine multimodal cues from multiple representations of light field. Three convolutional pathways are applied to predict the disparity label of each pixel from three modality representations of light field. Performing discrete disparity label classification instead of continuous disparity value regression allows us to easily fuse the prediction results of three pathways in the form of probability distribution. Evaluation experiments are carried out on the HCI 4D Light Field Benchmark. Our method, named as “FusionNet” on the benchmark website, ranks first among the published methods on high accuracy metrics of Bad Pixel Ratio 0.03 and 0.01.

A Fourier Disparity Layer Representation for Light Fields

In this paper, we present a new Light Field representation for efficient Light Field processing and rendering called Fourier Disparity Layers (FDL). The proposed FDL representation samples the Light Field in the depth (or equivalently the disparity) dimension by decomposing the scene as a discrete sum of layers. The layers can be constructed from various types of Light Field inputs, including a set of sub-aperture images, a focal stack, or even a combination of both. From our derivations in the Fourier domain, the layers are simply obtained by a regularized least square regression performed independently at each spatial frequency, which is efficiently parallelized in a GPU implementation. Our model is also used to derive a gradient descent-based calibration step that estimates the input view positions and an optimal set of disparity values required for the layer construction. Once the layers are known, they can be simply shifted and filtered to produce different viewpoints of the scene while controlling the focus and simulating a camera aperture of arbitrary shape and size. Our implementation in the Fourier domain allows real-time Light Field rendering. Finally, direct applications such as view interpolation or extrapolation and denoising are presented and evaluated.