4D Light Research Articles

Holographic Parallax Improves 3D Perceptual Realism

Holographic near-eye displays are a promising technology to solve long-standing challenges in virtual and augmented reality display systems. Over the last few years, many different computer-generated holography (CGH) algorithms have been proposed that are supervised by different types of target content, such as 2.5D RGB-depth maps, 3D focal stacks, and 4D light fields. It is unclear, however, what the perceptual implications are of the choice of algorithm and target content type. In this work, we build a perceptual testbed of a full-color, high-quality holographic near-eye display. Under natural viewing conditions, we examine the effects of various CGH supervision formats and conduct user studies to assess their perceptual impacts on 3D realism. Our results indicate that CGH algorithms designed for specific viewpoints exhibit noticeable deficiencies in achieving 3D realism. In contrast, holograms incorporating parallax cues consistently outperform other formats across different viewing conditions, including the center of the eyebox. This finding is particularly interesting and suggests that the inclusion of parallax cues in CGH rendering plays a crucial role in enhancing the overall quality of the holographic experience. This work represents an initial stride towards delivering a perceptually realistic 3D experience with holographic near-eye displays.

Take Your Model Further: A General Post-refinement Network for Light Field Disparity Estimation via BadPix Correction

Most existing light field (LF) disparity estimation algorithms focus on handling occlusion, texture-less or other areas that harm LF structure to improve accuracy, while ignoring other potential modeling ideas. In this paper, we propose a novel idea called Bad Pixel (BadPix) correction for method modeling, then implement a general post-refinement network for LF disparity estimation: Bad-pixel Correction Network (BpCNet). Given an initial disparity map generated by a specific algorithm, we assume that all BadPixs on it are in a small range. Then BpCNet is modeled as a fine-grained search strategy, and a more accurate result can be obtained by evaluating the consistency of LF images in this limited range. Due to the assumption and the consistency between input and output, BpCNet can perform as a general post-refinement network, and can work on almost all existing algorithms iteratively. We demonstrate the feasibility of our theory through extensive experiments, and achieve remarkable performance on the HCI 4D Light Field Benchmark.

Robust extension of light fields with probable 3D distribution based on iterative scene estimation from multi-focus images

In this study, we propose an iterative approach to perform pseudo-scene reconstruction based on multi-focus images for estimating the 4D light fields of the scene robustly. An analogous scene-independent linear filtering approach has been proposed to efficiently achieve free viewpoint image reconstruction and scene refocusing with virtual bokeh by solely extracting 3D frequency components. However, it only succeeded in synthesizing limited regions of 4D light fields from multi-focus images, which can lead to awful degradation of reconstructed free viewpoint images outside the range due to the lack of the corresponding frequency components. In this paper, we directly utilize a linear relationship of 3D convolution combining multi-focus images and 3D scene by 3D blurring filters to estimate the 3D distribution itself of the scene instead of deriving the scene-independent linear filters from the relationship. In other words, we perform the scene reconstruction as a simplified convex optimization problem by modeling multi-focus images in multi-layered blurring composed of the scene and a bokeh filter. Algebraic reconstruction techniques of computed tomography, which uses the classical steepest descent method combined with projection onto convex sets (POCS) to ensure the convexity of the problem constraints, are introduced into such 3D scene reconstruction. The new method can be highly anticipated to assist us in extensively estimating robust 4D light fields. Experimental results illustrate that this approach enables good quality 3D image reconstruction with a wider range of 4D light fields. We also discuss the reduction of high computational costs and enormous memory requirements imposed by iterative algebraic reconstruction techniques. The quality of the obtained 4D light fields is comprehensively evaluated using both synthetic and real multi-focus images of practicable image resolution and depth range, additionally to an analysis of the faster processing times when comparing our iterative approach to the performance of the prior linear filters.

Unsupervised disparity estimation from light field using plug-and-play weighted warping loss

We investigated disparity estimation from a light field using a convolutional neural network (CNN). Most of the methods implemented a supervised learning framework, where the predicted disparity map was compared directly to the corresponding ground-truth disparity map in the training stage. However, light field data accompanied with ground-truth disparity maps were insufficient and rarely available for real-world scenes. The lack of training data resulted in limited generality of the methods trained with them. To tackle this problem, we took a simple Figure-and-play approach to remake a supervised method into an unsupervised (self-supervised) one. We replaced the loss function of the original method with one that does not depend on the ground-truth disparity maps. More specifically, our loss function is designed to indirectly evaluate the accuracy of the disparity map by using warping errors among the input light field views. We designed pixel-wise weights to properly evaluate the warping errors in the presence of occlusions, and an edge loss to encourage edge alignment between the image and the disparity map. As a result of this unsupervised learning framework, our method can use more abundant training datasets (even those without ground-truth disparity maps) than the original supervised method. Our method was evaluated on computer-generated scenes (4D Light Field Benchmark) and real-world scenes captured by Lytro Illum cameras. Our method achieved the state-of-the-art performance as an unsupervised method on the benchmark. We also demonstrated that our method can estimate disparity maps more accurately than the original supervised method for various real-world scenes.

Image Quality Enhancement of 4d Light Field Microscopy Via Reference Image Propagation-Based One-Shot Learning

Image Quality Enhancement of 4d Light Field Microscopy Via Reference Image Propagation-Based One-Shot Learning

Attention-based Multi-Level Fusion Network for Light Field Depth Estimation

Depth estimation from Light Field (LF) images is a crucial basis for LF related applications. Since multiple views with abundant information are available, how to effectively fuse features of these views is a key point for accurate LF depth estimation. In this paper, we propose a novel attention-based multi-level fusion network. Combined with the four-branch structure, we design intra-branch fusion strategy and inter-branch fusion strategy to hierarchically fuse effective features from different views. By introducing the attention mechanism, features of views with less occlusions and richer textures are selected inside and between these branches to provide more effective information for depth estimation. The depth maps are finally estimated after further aggregation. Experimental results shows the proposed method achieves state-of-the-art performance in both quantitative and qualitative evaluation, which also ranks first in the commonly used HCI 4D Light Field Benchmark.

Deep residual pooling network for texture recognition

Current deep learning-based texture recognition methods extract spatial orderless features from pre-trained deep learning models that are trained on large-scale image datasets. These methods either produce high dimensional features or have multiple steps like dictionary learning, feature encoding and dimension reduction. In this paper, we propose a novel end-to-end learning framework that not only overcomes these limitations, but also demonstrates faster learning. The proposed framework incorporates a residual pooling layer consisting of a residual encoding module and an aggregation module. The residual encoder preserves the spatial information for improved feature learning and the aggregation module generates orderless feature for classification through a simple averaging. The feature has the lowest dimension among previous deep texture recognition approaches, yet it achieves state-of-the-art performance on benchmark texture recognition datasets such as FMD, DTD, 4D Light and one industry dataset used for metal surface anomaly detection. Additionally, the proposed method obtains comparable results on the MIT-Indoor scene recognition dataset. Our codes are available at https://github.com/maoshangbo/DRP-Texture-Recognition.

Overview of 4D Light Field Representation

Overview of 4D Light Field Representation

ALFO: Adaptive Light Field Over-Segmentation

Automatic image over-segmentation into superpixels has attracted increasing attention from researchers to apply it as a pre-processing step for several computer vision applications. In 4D Light Field (LF) imaging, image over-segmentation aims at achieving not only superpixel compactness and accuracy but also cross-view consistency. Due to the high dimensionality of 4D LF images, depth information can be estimated and exploited during the over-segmentation along with spatial and visual appearance features. However, balancing between several hybrid features to generate robust superpixels for different 4D LF images is challenging and not adequately solved in existing solutions. In this paper, an automatic, adaptive, and view-consistent LF over-segmentation method based on normalized LF cues and $K$ -means clustering is proposed. Initially, disparity maps for all LF views are estimated entirely to improve superpixel accuracy and consistency. Afterwards, by using $K$ -means clustering, a 4D LF image is iteratively divided into regular superpixels that adhere to object boundaries and ensure cross-view consistency. Our proposed method can automatically adjust the clustering weights of the various features that characterize each superpixel based on the image content. Quantitative and qualitative results on several 4D LF datasets demonstrate outperforming performance of the proposed method in terms of superpixel accuracy, shape regularity and view consistency when using adaptive clustering weights, compared to the state-of-the-art 4D LF over-segmentation methods.

Light-field-depth-estimation network based on epipolar geometry and image segmentation.

In this paper, we propose a convolutional neural network based on epipolar geometry and image segmentation for light-field depth estimation. Epipolar geometry is utilized to estimate the initial disparity map. Multi-orientation epipolar images are selected as input data, and the convolutional blocks are adopted based on the disparity of different-direction epipolar images. Image segmentation is used to obtain the edge information of the central sub-aperture image. By concatenating the output of the two parts, an accurate depth map could be generated with fast speed. Our method achieves a high rank on most quality assessment metrics in the HCI 4D Light Field Benchmark and also shows effectiveness in estimating accurate depth on real-world light-field images.

4D Light Field Segmentation From Light Field Super-Pixel Hypergraph Representation.

Efficient and accurate segmentation of full 4D light fields is an important task in computer vision and computer graphics. The massive volume and the redundancy of light fields make it an open challenge. In this article, we propose a novel light field hypergraph (LFHG) representation using the light field super-pixel (LFSP) for interactive light field segmentation. The LFSPs not only maintain the light field spatio-angular consistency, but also greatly contribute to the hypergraph coarsening. These advantages make LFSPs useful to improve segmentation performance. Based on the LFHG representation, we present an efficient light field segmentation algorithm via graph-cut optimization. Experimental results on both synthetic and real scene data demonstrate that our method outperforms state-of-the-art methods on the light field segmentation task with respect to both accuracy and efficiency.

3D-Kernel Foveated Rendering for Light Fields

Light fields capture both the spatial and angular rays, thus enabling free-viewpoint rendering and custom selection of the focal plane. Scientists can interactively explore pre-recorded microscopic light fields of organs, microbes, and neurons using virtual reality headsets. However, rendering high-resolution light fields at interactive frame rates requires a very high rate of texture sampling, which is challenging as the resolutions of light fields and displays continue to increase. In this article, we present an efficient algorithm to visualize 4D light fields with 3D-kernel foveated rendering (3D-KFR). The 3D-KFR scheme coupled with eye-tracking has the potential to accelerate the rendering of 4D depth-cued light fields dramatically. We have developed a perceptual model for foveated light fields by extending the KFR for the rendering of 3D meshes. On datasets of high-resolution microscopic light fields, we observe 3.47×-7.28× speedup in light field rendering with minimal perceptual loss of detail. We envision that 3D-KFR will reconcile the mutually conflicting goals of visual fidelity and rendering speed for interactive visualization of light fields.

Unsupervised Monocular Depth Estimation from Light Field Image.

Learning based depth estimation from light field has made significant progresses in recent years. However, most existing approaches are under the supervised framework, which requires vast quantities of ground-truth depth data for training. Furthermore, accurate depth maps of light field are hardly available except for a few synthetic datasets. In this paper, we exploit the multi-orientation epipolar geometry of light field and propose an unsupervised monocular depth estimation network. It predicts depth from the central view of light field without any ground-truth information. Inspired by the inherent depth cues and geometry constraints of light field, we then introduce three novel unsupervised loss functions: photometric loss, defocus loss and symmetry loss. We have evaluated our method on a public 4D light field synthetic dataset. As the first unsupervised method published in the 4D Light Field Benchmark website, our method can achieve satisfactory performance in most error metrics. Comparison experiments with two state-of-the-art unsupervised methods demonstrate the superiority of our method. We also prove the effectiveness and generality of our method on real-world light-field images.

A hybrid learning of multimodal cues for light field depth estimation

Learning based light field depth estimation methods have been proven to be successful, however, most of them focus on the depth cues extraction from a single representation of light field. In our opinion, existing methods are difficult to simultaneously learn various cues with different modalities, which are implied in different representations of light field. In this paper, we formulate the light field depth estimation as a pixel-wise classification task, and introduce a hybrid learning architecture to combine multimodal cues from multiple representations of light field. Three convolutional pathways are applied to predict the disparity label of each pixel from three modality representations of light field. Performing discrete disparity label classification instead of continuous disparity value regression allows us to easily fuse the prediction results of three pathways in the form of probability distribution. Evaluation experiments are carried out on the HCI 4D Light Field Benchmark. Our method, named as “FusionNet” on the benchmark website, ranks first among the published methods on high accuracy metrics of Bad Pixel Ratio 0.03 and 0.01.

Differential Scene Flow from Light Field Gradients

This paper presents novel techniques for recovering 3D dense scene flow, based on differential analysis of 4D light fields. The key enabling result is a per-ray linear equation, called the ray flow equation, that relates 3D scene flow to 4D light field gradients. The ray flow equation is invariant to 3D scene structure and applicable to a general class of scenes, but is under-constrained (3 unknowns per equation). Thus, additional constraints must be imposed to recover motion. We develop two families of scene flow algorithms by leveraging the structural similarity between ray flow and optical flow equations: local 'Lucas-Kanade' ray flow and global 'Horn-Schunck' ray flow, inspired by corresponding optical flow methods. We also develop a combined local-global method by utilizing the correspondence structure in the light fields. We demonstrate high precision 3D scene flow recovery for a wide range of scenarios, including rotation and non-rigid motion. We analyze the theoretical and practical performance limits of the proposed techniques via the light field structure tensor, a 3x3 matrix that encodes the local structure of light fields. We envision that the proposed analysis and algorithms will lead to design of future light-field cameras that are optimized for motion sensing, in addition to depth sensing.

A Concordance Analogue of the 4D Light Bulb Theorem

Abstract We prove a concordance analogue of Gabai’s $4$D light bulb theorem. That is, we show that when $R$ and $R^{\prime}$ are homotopic, embedded $2$-spheres in a $4$-manifold $X^4,$ where $\pi _1(X^4)$ has no $2$-torsion and one of $R$ or $R^{\prime}$ has a transverse sphere, then $R$ and $R^{\prime}$ are concordant. When $\pi _1(X^4)$ has $2$-torsion, we give a similar statement with extra hypotheses as in the $4$D light bulb theorem. We also give similar statements when $R$ and $R^{\prime}$ are orientable positive-genus surfaces.

Geometry-Aware Graph Transforms for Light Field Compact Representation.

The paper addresses the problem of energy compaction of dense 4D light fields by designing geometry-aware local graph-based transforms. Local graphs are constructed on super-rays that can be seen as a grouping of spatially and geometry-dependent angularly correlated pixels. Both non separable and separable transforms are considered. Despite the local support of limited size defined by the super-rays, the Laplacian matrix of the non separable graph remains of high dimension and its diagonalization to compute the transform eigen vectors remains computationally expensive. To solve this problem, we then perform the local spatio-angular transform in a separable manner. We show that when the shape of corresponding super-pixels in the different views is not isometric, the basis functions of the spatial transforms are not coherent, resulting in decreased correlation between spatial transform coefficients. We hence propose a novel transform optimization method that aims at preserving angular correlation even when the shapes of the super-pixels are not isometric. Experimental results show the benefit of the approach in terms of energy compaction. A coding scheme is also described to assess the rate-distortion perfomances of the proposed transforms and is compared to state of the art encoders namely HEVC-lozenge [1], JPEG pleno 1.1 [2], HEVC-pseudo [3] and HLRA [4].

4D Light Field Superpixel and Segmentation.

Superpixel segmentation of 2D images has been widely used in many computer vision tasks. Previous algorithms model the color, position, or higher spectral information for segmenting a 2D image. However, limited to the Gaussian imaging principle in a traditional camera, where each pixel is formed by summing lots of light rays from different angles, there is not a thorough segmentation solution to eliminate the ambiguity in defocus and occlusion boundary areas. In this paper, we consider the essential element of image pixel, i.e., rays in light space, and propose light field superpixel (LFSP) to eliminate the ambiguity. The LFSP is first defined mathematically and then two evaluation metrics, named LFSP self-similarity and effective label ratio, are proposed to evaluate the refocus-invariant and full-sliced properties of segmentation. By building a clique system containing 80 neighbors in light field, a robust refocus-invariant LFSP segmentation algorithm is developed. Experimental results on both synthetic and real light field datasets demonstrate the advantages over the current state of the art in terms of traditional evaluation metrics. Additionally, the LFSP self-similarity evaluations under different light field refocus levels show the refocus-invariance of the proposed algorithm. The full-sliced property of the proposed LFSP algorithm is verified by comparing it with the classical supervoxel algorithms. Finally, an LFSP-based application is demonstrated to show the effectiveness of LFSP in light field editing.

Highly parallel steered mixture-of-experts rendering at pixel-level for image and light field data

A novel image approximation framework called steered mixture-of-experts (SMoE) was recently presented. SMoE has multiple applications in coding, scale-conversion, and general processing of image modalities. In particular, it has strong potential for coding and streaming higher dimensional image modalities that are necessary to leverage full translational and rotational freedom (6 degrees-of-freedom) in virtual reality for camera captured images. In this paper, we analyze the rendering performance of SMoE for 2D images and 4D light fields. Two different GPU implementations that parallelize the SMoE regression step at pixel-level are presented, including experimental evaluations based on rendering performance and quality. In this paper it is shown that on appropriate hardware, an OpenCL implementation can achieve 85 fps and 22 fps for, respectively, 1080p and 4K renderings of large models with more than 100,000 of Gaussian kernels.

Construction of a flexible and scalable 4D light field camera array using Raspberry Pi clusters

The light field is a 4D function that represents the radiance of light traveling in every direction through every point in 3D space. In this paper, we demonstrate how effectively 4D light fields can be sampled from the real world using a custom-built 4D light field camera system constructed based on cheaply and commonly available Raspberry Pi computers. The camera system consisting of a camera array and a dedicated calibration software is flexible and scalable in structure because the system can be reconfigured easily to meet the user’s needs. We show the effectiveness of the camera system by interactively visualizing captured 4D light fields using our GPU-accelerated light field renderer supporting a virtual camera model that synthesizes various photographic effects including zooming and panning, refocusing, focus breathing, and depth-of-field control. We believe that the do-it-yourself 4D light field camera will be explored effectively in traditional application fields, such as computer graphics and vision, and in other areas like science education.