Light Field Information Research Articles

Improvements of resolution of light field imaging based on four-dimensional optical framing via a semi-transparent mirror.

A simple light field imaging system is proposed, which could improve the resolution of light field imaging and enhance the signal-to-noise ratio of the result image. In the experiment, the light field imaging system consists of a single CCD with a microlens array and a semi-transparent mirror. The Fourier slice theorem has been used to transform the four-dimensional (4D) light field information into an infinite number of two-dimensional (2D) slices. With the use of the semi-transparent mirror, the high spatial resolution image can be reconstructed on the terminal sensor. The proposed method can not only reduce the aliasing defocus degree in the imaging process but also improve the slice image resolution to meet the requirement of image definition.

Refocusing Algorithm Based on Nonlinear SAI System

Synthetic aperture imaging (SAI) technology gets the light field information of the scene through the camera array. With the large virtual aperture, it can effectively acquire the information of the partially occluded object in the scene, and then we can focus on the arbitrary target plane corresponding to the reference perspective through the refocus algorithm. Meanwhile, objects that deviate from the plane will be blurred to varying degrees. However, when the object to be reconstructed in the scene is occluded by the complex foreground, the optical field information of the target cannot be effectively detected due to the limitation of the linear array. In order to deal with these problems, this paper proposes a nonlinear SAI method. This method can obtain the occluded object’s light field information reliably by using the nonlinear array. Experiments are designed for the nonlinear SAI, and refocusing is performed for the occluded objects with different camera arrays, different depths, and different distribution intervals. The results demonstrate that the method proposed in this paper is advanced than the traditional SAI method based on linear array.

Multiple aperture underwater imaging algorithm based on polarization information fusion

Underwater optical imaging is the key technology to explore the underwater mystery. However, due to the absorption and backscattering effects of the media in the underwater environment, the image acquired by the detector will be severely degraded. In order to obtain the effective underwater scene information, it is necessary to restore the acquired underwater image. The restoration technology based on differential polarization is one of the main methods of restoring the underwater images, which can suppress the background scattered light by the common-mode suppression between orthogonal polarization graphs, thus realizing the restoration of underwater image. However, the relevant research shows that the restoration effect of this method is general for the underwater non-uniform light field. The main reason is that the estimation errors of polarization degree and background scattering intensity under the condition of the non-uniform underwater light field are large. Out of the above problem, in this paper we present the multiple aperture underwater imaging technology of fused polarization information. The method uses the camera array to realize the large virtual aperture imaging system, thus obtaining the wide-angle light field information, and then to fuse the depth information of the scene to realize the accurate estimation of background scattering light intensity and polarization degree under the condition of underwater non-uniform light field. The estimated parameter value can better reflect the global characteristics of the scene. Through the imaging experiments on the targets with different polarization degrees in the turbid underwater environment, comparing with the current advanced restoration algorithm, the results show that the proposed method can effectively solve the problems of background scattering and polarization degree significant estimation error caused by non-uniform underwater light field, and obtain high-quality restoration results. Through the contrast imaging experiment of the target in the underwater environment with different turbidity concentrations, the results show that with the increase of turbidity concentration of the water, the image recovery effect of the method in this paper is gradually weakened. However, it still has a good restoration effect at a large concentration. At the same time, imaging experiments are conducted on targets in underwater environments with different sediment concentrations. The results show that the method proposed in this paper can also obtain a better restoration image in the turbid water environment containing sediment.

LFNet: Light Field Fusion Network for Salient Object Detection.

In this work, we propose a novel light field fusion network-LFNet, a CNNs-based light field saliency model using 4D light field data containing abundant spatial and contextual information. The proposed method can reliably locate and identify salient objects even in a complex scene. Our LFNet contains a light field refinement module (LFRM) and a light field integration module (LFIM) which can fully refine and integrate focusness, depths and objectness cues from light field image. The LFRM learns the light field residual between light field and RGB images for refining features with useful light field cues, and then the LFIM weights each refined light field feature and learns spatial correlation between them to predict saliency maps. Our method can take full advantage of light field information and achieve excellent performance especially in complex scenes, e.g., similar foreground and background, multiple or transparent objects and low-contrast environment. Experiments show our method outperforms the state-of-the-art 2D, 3D and 4D methods across three light field datasets.

Three-dimensional measurement of bubble in gas-liquid flow by light field photography

A three-dimensional measurement method for bubble parameters in gas-liquid two-phase flow was proposed based on the light field photography for solving the problem that bubble parameters could only be measured in two-dimensional problems using conventional photography. The light field camera recorded the light field information of the gas-liquid two-phase flow, and then the total focus image and the refocused images of the bubbles in the two-phase flow field were obtained using a computational imaging technique. The image processing was performed by binarizing the total focus image to obtain the projection of bubbles in the depth direction and evaluating the sharpness of refocused images to obtain the depth of bubbles. Based on the bubble projection and depth information, the three-dimensional reconstruction of the bubbles was achieved, and the parameters, such as size distribution, spatial position and gas void fraction of bubbles were further calculated. The calibration and the experiment were made to evaluate the proposed method. Experimental results show that the cross-sectional void fraction in the depth direction is in agreement with the bubble depth distribution and oscillates periodically in the vertical direction. With the increase of gas flow rates from 0.5 L/min to 1.1 L/min, the amount of bubbles increases from 28 to 72; the average diameter of bubbles maintains about 1.56 mm before the gas flow rate reaches 0.9 L/min, after increasing sharply; the gas volume fraction rises approximately linearly, and the bubble depth interval increases gradually. The three-dimensional reconstruction results of bubbles are in good agreement with the law of bubble formation and distribution, thus verifying the feasibility of the proposed method.

3-D Computational Ghost Imaging With Extended Depth of Field for Measurement

The accuracy of traditional 3-D measurements is limited by the depth of field (DOF), such as in binocular imaging and holography. Ghost imaging (GI) can achieve lensless imaging with a DOF deeper than the traditional imaging. To realize this, a computational GI method is proposed and a system built, in which the light field information and modulation information are used to reconstruct the 3-D image. This system uses two projection devices for light modulation and pattern projecting, and utilizes a detector to receive the reflected signals of the two projectors on the object synchronously. The experiment shows that for an original DOF of 0.4 m, our system can extend the depth to 1.5 m. The method should have the great application prospects in the fields of remote sensing, imaging guidance, and medical imaging.

Light Field Super-Resolution Using Edge-Preserved Graph-Based Regularization

The light field information would be captured through light field cameras and in different directions regarding 3D image view recordings. In this paper, in order to increase the spatio-angular super-resolution quality and to decrease the reconstruction error regarding the light field information, we use a graph-based light field super-resolution strategy. Accordingly, in order to apply the complementary data in the light field views, we use a graph regularizer for the total recovery of the information and an edge-preserving technique that represents an isometry between curves in the 2D manifold and 5D space of the RGB image views. Moreover, the reconstruction of the light field information is based on applying the alternating direction method of multipliers (ADMM) algorithm. Accordingly, a recent enhanced ADMM model has been used in this paper which is denominated as “Plug-and-Play” and permits the user to plug an image reconstruction technique and a denoising methodology as the first and second sub-problems respectively. On that account, we would be able to resolve the light field super-resolution problem considering the graph-based light field structure as the first sub-problem and the edge-preserving technique as the denoising methodology. Consequently, by applying the proposed super-resolution strategy, the super-resolved light field outcome would be more favorable in terms of visual quality and reconstruction errors in comparison with other state-of-the-art methodologies.

Fourier light-field microscopy.

Observing the various anatomical and functional information that spans many spatiotemporal scales with high resolution provides deep understandings of the fundamentals of biological systems. Light-field microscopy (LFM) has recently emerged as a scanning-free, scalable method that allows for high-speed, volumetric imaging ranging from single-cell specimens to the mammalian brain. However, the prohibitive reconstruction artifacts and severe computational cost have thus far limited broader applications of LFM. To address the challenge, in this work, we report Fourier LFM (FLFM), a system that processes the light-field information through the Fourier domain. We established a complete theoretical and algorithmic framework that describes light propagation, image formation and system characterization of FLFM. Compared with conventional LFM, FLFM fundamentally mitigates the artifacts, allowing high-resolution imaging across a two- to three-fold extended depth. In addition, the system substantially reduces the reconstruction time by roughly two orders of magnitude. FLFM was validated by high-resolution, artifact-free imaging of various caliber and biological samples. Furthermore, we proposed a generic design principle for FLFM, as a highly scalable method to meet broader imaging needs across various spatial levels. We anticipate FLFM to be a particularly powerful tool for imaging diverse phenotypic and functional information, spanning broad molecular, cellular and tissue systems.

A broadband achromatic metalens array for integral imaging in the visible

Integral imaging is a promising three-dimensional (3D) imaging technique that captures and reconstructs light field information. Microlens arrays are usually used for the reconstruction process to display 3D scenes to the viewer. However, the inherent chromatic aberration of the microlens array reduces the viewing quality, and thus, broadband achromatic imaging remains a challenge for integral imaging. Here, we realize a silicon nitride metalens array in the visible region that can be used to reconstruct 3D optical scenes in the achromatic integral imaging for white light. The metalens array contains 60 × 60 polarization-insensitive metalenses with nearly diffraction-limited focusing. The nanoposts in each high-efficiency (measured as 47% on average) metalens are delicately designed with zero effective material dispersion and an effective achromatic refractive index distribution from 430 to 780 nm. In addition, such an achromatic metalens array is composed of only a single silicon nitride layer with an ultrathin thickness of 400 nm, making the array suitable for on-chip hybrid-CMOS integration and the parallel manipulation of optoelectronic information. We expect these findings to provide possibilities for full-color and aberration-free integral imaging, and we envision that the proposed approach may be potentially applicable in the fields of high-power microlithography, high-precision wavefront sensors, virtual/augmented reality and 3D imaging.

Underwater Imaging Based on LF and Polarization

The underwater image restoration based on polarization information has achieved good results in improving the image quality in scattering media. However, the previous methods are difficult to obtain the true distribution of degree of polarization in the scene. In this paper, we combine synthetic aperture imaging with polarimetric imaging, and propose a method for retrieving radiation of object based on the degree of polarization and intensity of backscattering at the multi-view image. In addition, compared with the previous methods, the proposed method can achieve simultaneous acquisition of four-dimensional light field information and polarization information, effectively increasing the information dimension obtained by single imaging. In order to verify the effectiveness and superiority of the proposed method, we have established a relevant experimental platform and compared with the experimental results of the previous methods, and obtained the expected experimental results.

Achromatic metalens array for full-colour light-field imaging.

A light-field camera captures both the intensity and the direction of incoming light1-5. This enables a user to refocus pictures and afterwards reconstruct information on the depth of field. Research on light-field imaging can be divided into two components: acquisition and rendering. Microlens arrays have been used for acquisition, but obtaining broadband achromatic images with no spherical aberration remains challenging. Here, we describe a metalens array made of gallium nitride (GaN) nanoantennas6 that can be used to capture light-field information and demonstrate a full-colour light-field camera devoid of chromatic aberration. The metalens array contains an array of 60 × 60 metalenses with diameters of 21.65 μm. The camera has a diffraction-limited resolution of 1.95 μm under white light illumination. The depth of every object in the scene can be reconstructed slice by slice from a series of rendered images with different depths of focus. Full-colour, achromatic light-field cameras could find applications in a variety of fields such as robotic vision, self-driving vehicles and virtual and augmented reality.

Defocusing mechanism and focusing evaluation function of light field imaging

The light field imaging technology can realize the application of the full-focus image synthesis of the scene and the de-occlusion reconstruction of background target through the classification of the light field information of the scene. How to effectively evaluate the focus area in the image is a prerequisite for the above application. The dispersion characteristics of the defocusing point of a conventional imaging system are fundamentally different from those of the defocusing area in the refocusing image of the light field. Therefore, the evaluation criteria based on the diffusion characteristics of the defocusing point cannot be applied to the evaluation of the focus of the refocusing image of the light field. Aiming at the above problems, in this paper, starting from the principle of refocusing of light field imaging, we analyze the blurring characteristics of the defocused target image, and propose a new evaluation function of the focus of the refocusing image of the light field. Based on this, the refinement segmentation method of the focal region is studied to achieve the final focus area extraction. According to the indoor scene data set captured by the camera array of Stanford university, in this paper we use the traditional focus degree evaluation algorithm and the algorithm to evaluate the focusing degree of the foreground target potted plant in the scene and obtain the complete information about the foreground target, therefore we also study the refined segmentation algorithm. Then, in the process of refocusing the background object (CD box), the foreground light is screened out, and the reconstructed image of the specified focusing plane is obtained. Using the peak-signal-to-noise ratio and mean structural similarity index measure to evaluate the quality of the target in refocusing area, the results show that the proposed algorithm in this paper can effectively mark and separate the imaginary artifact information and ensure the high-quality focus reconstruction of the partially occluded target in the scene, which can effectively overcome the influence of the edge and texture information of the object in the scene on the defocusing area. The method presented in this work has better adaptability to the focus degree evaluation of the refocusing image of the light field.

Research on high-resolution imaging technology based on light field manipulation for a lenslet-based plenoptic camera.

In this paper, an aberration correction method for an extended target is proposed to solve the problem of the lenslet-based plenoptic camera not imaging clearly under the influence of aberrations. We propose a light field manipulation method to improve performance of the light field imaging system. The principle of this method is that the sub-aperture images extracted from the raw light field image are offset when the light field imaging system is affected by aberrations, and the symmetrical arrangement of the sub-aperture image array is destroyed. By repairing the symmetrical arrangement of the sub-aperture image array, the influence of phase aberrations on the imaging system can be eliminated, and the resolution of the plenoptic camera can be improved. We use an image correlation algorithm to process the sub-aperture images of the plenoptic camera, calculate and compensate each sub-aperture image's displacement caused by aberrations, and restore the symmetrical arrangement of the sub-aperture image array; then, a corrected high-resolution refocused image can be generated. In particular, this method uses only the raw light field information obtained by the plenoptic camera in a single exposure, without adding other hardware devices. Furthermore, it takes the extended target itself as the reference image, so the ideal position need not be calibrated in advance. Also, the parallax information of the sub-aperture images is retained, and the method is simple and easy to use. Numerical simulation and experimental results show that the technology proposed in this paper can work well for high-resolution imaging of a plenoptic camera with phase aberrations. This method can be potentially applied to analyze lens aberration, media-induced image distortion such as water turbulence in underwater imaging, and atmospheric turbulence in remote imaging. It may have important application prospects in the fields of astronomical object detection, remote sensing, etc.

Projection-Type Multiview Holographic Three-Dimensional Display Using a Single Spatial Light Modulator and a Directional Diffractive Device

In this paper, we propose a projection-type multiview holographic three-dimensional (3-D) display using a single spatial light modulator (SLM) and a directional diffractive device, where a phase-only Fresnel hologram is calculated and displayed on the SLM to reconstruct the multiview hybrid image of a 3-D object for providing the amplitude information of a multiview light field, and a directional diffractive device covered with many pixelated gratings is designed to guide different reconstructed perspectives into different viewing zones for expressing the phase information of the light field. Furthermore, a four-view directional diffractive device is fabricated by using the pixelated holographic exposure method, and its optical properties are also measured under a collimated illumination of the green reconstruction laser with the designed wavelength. In addition, an experimental verification system is constructed to demonstrate a four-view 3-D display. The recorded experimental results confirm that a crosstalk-free four-view light field is created, which is beneficial from the precise alignment between the directional diffractive device and the reconstructed image. Moreover, the capability of dynamic display with the proposed system is also tested by refreshing the SLM with the sequential holograms, and the captured videos show a promising potential for the real-time multiview holographic 3-D projectors.

Rectification of Images Distorted by Microlens Array Errors in Plenoptic Cameras.

A plenoptic cameras is a sensor that records the 4D light-field distribution of target scenes. The surface errors of a microlens array (MLA) can cause the degradation and distortion of the raw image captured by a plenoptic camera, resulting in the confusion or loss of light-field information. To address this issue, we propose a method for the local rectification of distorted images using white light-field images. The method consists of microlens center calibration, geometric rectification, and grayscale rectification. The scope of its application to different sized errors and the rectification accuracy of three basic surface errors, including the overall accuracy and the local accuracy, are analyzed through simulation of imaging experiments. The rectified images have a significant improvement in quality, demonstrating the provision of precise light-field data for reconstruction of real objects.

One Shot 360-Degree Light Field Capture and Reconstruction with Depth Extraction Based on Optical Flow for Light Field Camera

A system for capturing 360-degree light field information of real-existing object in one shot and then optically reconstructing is proposed. A new depth extraction algorithm for light field cameras is proposed and various camera specifications for practical use of the algorithm are employed for the analysis. With a depth extraction method that was based on optical flow for light field camera, the depth information is extracted more accurately, according to the various specifications of light field camera. For 360-degree shooting, a simple capturing system composed of two mirrors and a light field camera is used. The capturing system has an advantage of being compact and inexpensive. The locations and orientations of the two mirrors are analyzed to optimize 360-degree light field recording. Holographic display is used to optically reconstruct the captured light field information. Experimental and simulation results are presented to support the proposed system and analysis.

Local error and its identification for microlens array in plenoptic camera

A microlens array (MLA) is a key optical element in light-field imaging, but surface errors caused by manufacturing defects can result in the loss and deviation of light-field information transmission. To address this issue, we establish a local error model for an MLA arranged in a matrix form, and develop a method for identifying error microlenses based on image quality evaluation indexes. The local imaging characteristics and degradation mechanism of three basic errors are analyzed through simulation. The local errors on different microlenses cause different degradations of the corresponding sub-images, and the trend of change in light-field image quality is related to the error type, error value, and error direction. The simulation results also verify the accuracy and effectiveness of the proposed method and models.

Research on computationally adaptive plenoptic imaging

As for computational adaptive plenoptic imaging system, the light-field of the target and interference are measured together, and then according to distribution characteristics of the four-dimensional light-field information between the target and the disturbed factors, target and disturbed factors can be effectively separated. This technique can be used to detect and recover the wavefront distortion caused by interference in the large field of view, and adaptively compensate for complicated wavefront aberration by means of computation. Compared with the traditional adaptive optics imaging method, the proposed method has a larger detecting field of view, and can directly analyze and compute wavefront information based on the extended target.

Demonstration of a large-size horizontal light-field display based on the LED panel and the micro-pinhole unit array

A 54-inch horizontal-parallax only light-field display based on the light-emitting diode (LED) panel and the micro-pinhole unit array (MPUA) is demonstrated. Normally, the perceived 3D effect of the three-dimensional (3D) display with smooth motion parallax and abundant light-field information can be enhanced with increasing the density of viewpoints. However, the density of viewpoints is inversely proportional to the spatial display resolution for the conventional integral imaging. Here, a special MPUA is designed and fabricated, and the displayed 3D scene constructed by the proposed horizontal light-field display is presented. Compared with the conventional integral imaging, both the density of horizontal viewpoints and the spatial display resolution are significantly improved. In the experiment, A 54-inch horizontal light-field display with 42.8° viewing angle based on the LED panel with the resolution of 1280 × 720 and the MPUA is realized, which can provide natural 3D visual effect to observers with high quality.

Analyzing the impact of the initialization for the nonnegative tensor factorization in tensor displays

Tensor display is an option in glasses-free three-dimensional (3-D) display technology. An initial solution has to be set to decompose the light-field information to be represented by the system. We have analyzed the impact of the initial guess on the multiplicative update rules in terms of peak signal-to-noise ratio, and proposed a method based on depth map estimation from an input light field. Results from simulations were obtained and compared with previous literature. In our sample, the initial values used have a large influence on results and convergence to a local minimum. The quality of the output stabilizes after a certain number of iterations, suggesting that a limit on such numbers should be imposed. We show that the proposed methods outperform the pre-existing ones.