Four-dimensional Light Field Research Articles

Quantitative real-time phase microscopy for extended depth-of-field imaging based on the 3D single-shot differential phase contrast (ssDPC) imaging method.

Optical diffraction tomography (ODT) is a promising label-free imaging method capable of quantitatively measuring the three-dimensional (3D) refractive index distribution of transparent samples. In recent years, partially coherent ODT (PC-ODT) has attracted increasing attention due to its system simplicity and absence of laser speckle noise. Quantitative phase imaging (QPI) technologies represented by Fourier ptychographic microscopy (FPM), differential phase contrast (DPC) imaging and intensity diffraction tomography (IDT) need to collect several or hundreds of intensity images, which usually introduce motion artifacts when shooting fast-moving targets, leading to a decrease in image quality. Hence, a quantitative real-time phase microscopy (qRPM) for extended depth of field (DOF) imaging based on 3D single-shot differential phase contrast (ssDPC) imaging method is proposed in this research study. qRPM incorporates a microlens array (MLA) to simultaneously collect spatial information and angular information. In subsequent optical information processing, a deconvolution method is used to obtain intensity stacks under different illumination angles in a raw light field image. Importing the obtained intensity stack into the 3D DPC imaging model is able to finally obtain the 3D refractive index distribution. The captured four-dimensional light field information enables the reconstruction of 3D information in a single snapshot and extending the DOF of qRPM. The imaging capability of the proposed qRPM system is experimental verified on different samples, achieve single-exposure 3D label-free imaging with an extended DOF for 160 µm which is nearly 30 times higher than the traditional microscope system.

3D salient object detection based on light field integral imaging.

Potent usage of the multi-scale light field information for salient object detection (SOD) is the essential requirement of three-dimensional (3D) SOD. On this basis, a light field 3D-SOD scheme is proposed that employs the pixel mapping algorithm to achieve a more distinct representation of spatial and angular information in the four-dimensional (4D) light field, collaboratively mining the global saliency cues via the co-salient object detection (CoSOD) network. Compared with the previous method, our scheme filters out most of the noise by thoroughly leveraging the global dependence of the 4D light field, offering significant enhancements in saliency extraction performance and efficiency. Additionally, the 3D reconstruction results demonstrate the integral retention of the spatial and angular information of the original light field.

Harnessing disordered photonics via multi-task learning towards intelligent four-dimensional light field sensors

The complete description of a continuous-wave light field includes its four fundamental properties: wavelength, polarization, phase and amplitude. However, the simultaneous measurement of a multi-dimensional light field of such four degrees of freedom is challenging in conventional optical systems requiring a cascade of dispersive and polarization elements. In this work, we demonstrate a disordered-photonics-assisted intelligent four-dimensional light field sensor. This is achieved by discovering that the speckle patterns, generated from light scattering in a disordered medium, are intrinsically sensitive to a high-dimension light field given their high structural degrees of freedom. Further, the multi-task-learning deep neural network is leveraged to process the single-shot light-field-encoded speckle images free from any prior knowledge of the complex disordered structures and realizes the high-accuracy recognition of full-Stokes vector, multiple orbital angular momentum (OAM), wavelength and power. The proof-of-concept study shows that the states space of four-dimensional light field spanning as high as 1680=4 (multiple-OAM) times2 (OAM power spectra) times15 (multiple-wavelength) times14 (polarizations) can be well recognized with high accuracy in the chip-integrated sensor. Our work provides a novel paradigm for the design of optical sensors for high-dimension light fields, which can be widely applied in optical communication, holography, and imaging.

Geometric Parameters Calibration of Focused Light Field Camera Based on Edge Spread Information Fitting

In this paper, a novel method based on edge spread information fitting (ESIF) is proposed to accurately calibrate the geometric parameters of a focused light field camera. A focused light field camera with flexible adjustment of spatial resolution and angular resolution is designed and built to capture the four-dimensional light field information of the scenes, and the geometric relationship between the focus plane of the camera and its internal parameters is derived to establish and simplify the calibration model. After that, a new algorithm based on sub-pixel edge fitting is designed to accurately detect corners, and the minimum corner size can be calculated to confirm the focus plane. In the simulation experiments, the error is 0.083% between the ground truth and the calibration result. The physical experiments show that our method is effective and reliable for the geometric calibration of a focused light field camera. Our method cleverly utilizes the special imaging geometric relationship of the focused light field camera to ensure the better calibration accuracy, which makes the calibration process more reliable.

A dense light field reconstruction algorithm for four-dimensional optical flow constraint equation

Dense light field sampling is an important basis for refocusing, depth estimation and 3-D imaging. It is difficult to obtain high resolution dense light field with a large-scale camera array and expensive equipment. At the same time, the current storage devices and transmission bandwidth also limit this technology's post-processing and application. In order to effectively reconstruct the angle domain of the light field based on the sparse light field data, this paper analyzes the correlation and constraint relationship between the optical flow field and the motion field of multi-view images in the same scene, extends the traditional optical flow constraint equation of two-dimensional imaging to the optical flow constraint equation of four-dimensional light field, and establishes an effective mathematical model. The coordinate position of the original pixel in the new angle image is determined by coordinate search, and its intensity is reconstructed. The experimental results of multi-scene dense reconstruction show that the proposed method can reconstruct the texture, shadow and color information in the light field of a long-baseline scene with high quality. The quantitative evaluation results show that the algorithm can be applied to dense light field reconstruction of complex scenes. The algorithm in this paper is only suitable for the case of optical flow constraint in a linear light field, and the follow-up research will focus on the case of nonlinear optical flow constraint.

Calculation model and analysis of fragment position with light field imaging mechanism under considering engineering uncertainty error factors

The spatial positions of multiple fragments formed by projectile explosion have random uncertainty dispersion state, so it is difficult to recognize each fragment and measure its spatial position when multiple fragments reach a certain area at the same time. In order to solve the problem, we propose a test method for measuring the spatial positions of multiple fragments based on dual light field intersection imaging mechanism, study refocusing principle of micro-lens array and four-dimensional light field imaging information, and establish the analytical model of spatial positions of multiple fragments. We set up the uncertainty error evaluation function of two light field cameras intersection test method from the perspective of engineering design, and give the uncertainty analysis method under engineering model. The influence of various factors on the uncertainty of test results under the engineering model is analyzed by calculation. Combined with the actual test data of the wooden target, the results show that the fragment spatial position test method based on dual light field intersection imaging mechanism is feasible and scientific.

Research on the Immersive Classroom Teaching Mode of Colleges and Universities Based on Virtual Reality

In order to improve the teaching effect of colleges and universities and eliminate the disadvantage Figures of the traditional teaching mode, this paper applies virtual reality technology to the immersive teaching mode of colleges and universities and uses four-dimensional light field parameterization to describe the acquisition process and reproduction process of integrated imaging. Moreover, this paper studies the propagation characteristics of the light field during the reproduction of the integrated imaging system and establishes an evaluation method for the 3D scene reproduction capability of the integrated imaging system using light field parameters. In addition, this paper analyzes the problem of parameter matching between the acquisition end and the display end in the current integrated imaging system and does a detailed study on the method of light field conversion and gives the interpolation method of the light field image. Finally, this paper constructs an immersive teaching system for colleges and universities based on virtual reality and verifies the performance of the system through experimental research. From the experimental research results, it can be seen that the immersive classroom teaching model in colleges and universities based on virtual reality proposed in this paper has good teaching effects.

Underwater large field of view 3D imaging based on fisheye lens

As for the 3D imaging of seabed topography, seabed sunken ships and large underwater buildings, the field of view is a significant technical index of the 3D imaging system. In this paper, a binocular vision imaging system based on fisheye lens is applied to satisfy the requirements for 3D imaging of large underwater scenes. Considering the severe image distortion caused by the large wide-angle of fisheye lens and the failure of pinhole imaging model underwater, the imaging process of underwater fisheye camera is re-described through the parametric representation of four-dimensional light field, and the multi-layer refraction imaging model of underwater binocular stereo vision is established based on fisheye camera. A conversion imaging model is constructed to convert the corrected underwater fisheye image into the equivalent image taken in the air. Combined with the existing stereo matching methods, the converted images realize the 3D reconstruction of a large-scale underwater scene. Experimental results show that the corrected underwater fisheye images have a small average reprojection error, which evidences the effectiveness of the proposed calibration method. In the experiment of 3D reconstruction for the large scene composed of multiple underwater ceramic discs, the reconstructed targets have good 3D imaging shapes, which indicates the developed underwater binocular fisheye camera imaging system has the ability to perform 3D imaging and make observations of large underwater scenes.

Lossless Coding of Light Fields Based on 4D Minimum Rate Predictors.

Common representations of light fields use four-dimensional data structures, where a given pixel is closely related not only to its spatial neighbours within the same view, but also to its angular neighbours, co-located in adjacent views. Such structure presents increased redundancy between pixels, when compared with regular single-view images. Then, these redundancies are exploited to obtain compressed representations of the light field, using prediction algorithms specifically tailored to estimate pixel values based on both spatial and angular references. This paper proposes new encoding schemes which take advantage of the four-dimensional light field data structures to improve the coding performance of Minimum Rate Predictors. The proposed methods expand previous research on lossless coding beyond the current state-of-the-art. The experimental results, obtained using both traditional datasets and others more challenging, show bit-rate savings no smaller than 10%, when compared with existing methods for lossless light field compression.

Measurement model and method of multiple projectile dispersion position based on dual light field intersection imaging

In order to solve the problem of measuring the spatial position of each projectile that are fired using multi-barrel weapons in high-frequency continuous firing modes, this paper proposes a new method to measure the spatial position of multiple projectiles with dual light field intersection imaging mechanism. According to the four-dimensional light field imaging information of light field camera and its micro-lens refocusing principle, we establish the calculation model of the spatial position of multiple projectiles; research the spatial geometric relationship of multiple projectiles in the dual light field intersection imaging system, analyze the light transfer mechanism of projectiles image by using the relation of main lens, micro-lens array and image sensor, and derive density distribution function of multiple projectiles; give the error calculation function of projectiles spatial position by differential method. Through calculation and comparative tests, the results verify the scientificity and rationality of the proposed calculation model.

Pseudo Video and Refocused Images-Based Blind Light Field Image Quality Assessment

The commercial light field camera is able to capture four-dimensional Light Field Image (LFI), which can be visualized to LFI contents on 2D displays by means of the Pseudo Video (PV) or the Refocused Images (RIs) generated with the refocusing function of LFI. However, the quality degradation of LFI will affect user’s visual experience of LFI contents. Hence, it is crucial to develop an effective LFI quality assessment method to monitor the LFI quality. Most existing subjective databases of LFI use PV and RIs visualization techniques to assess the quality of LFI. Therefore, as the way of presenting LFI on 2D display, PV and RIs are closely related to the subjective perception of LFI by human eyes. Based on these two visualization techniques, this article proposes a novel PV and RIs based blind LFI quality assessment method, in which the feature extraction is divided into two parts. In the first part, the PV’s structure, motion and disparity information are extracted with multi-scale and multi-directional Shearlet transform. In the other part, the spatial structure, depth and semantic information of the RIs are obtained. Finally, support vector regression is used to nonlinear map the perceptual features to quality score of LFI. The experimental results on four LFI databases show that the proposed method has better correlation with human visual perception, compared with the classical 2D image quality assessment methods as well as the state-of-the-art LFI quality assessment methods.

Wave-digital filter circuits for single-chip 4-D light field depth-based enhancement

In four-dimensional (4-D) light field (LF) processing, 4-D linear shift-invariant filters having hyperplanar passbands are used for depth-based scene enhancement. In this paper, two low-sensitivity and low-complexity field programmable gate array (FPGA)-based digital hardware architectures for 4-D hyperplanar filters are proposed for on-chip real-time processing of LFs. Both 4-D filters are designed exploiting resonant properties of multi-dimensional passive prototype networks, and are realized as wave digital filters (WDFs). The two 4-D WDF realizations are implemented as raster-scanned processing architectures on a Xilinx Virtex 6 Sx35 FPGA with a real-time clock speed of up to 33 MHz. This corresponds to a real-time throughout of 16.8 LFs/s for an LF of size $$11 \times 11 \times 128 \times 128$$ .

A Deep Learning Based Spatial Super-Resolution Approach for Light Field Content

One of the main issues encountered when working with Light Field technology is the trade-off between spatial and angular resolution. Various approaches have been presented to super-resolve light field in the spatial dimension. The challenge with current techniques is that they underuse the full potential of the information in the light field, as they process each sub-aperture image separately and do not consider disparity consistency. In this paper, we present a novel method for Light Field spatial super-resolution using the full four-dimensional light field spatial and angular information. We propose a learning-based model for spatial super-resolution which considers simultaneously all the sub-aperture images and also takes advantage of the epipolar image plane (EPI) information to ensure smooth disparity between the generated views and in turn construct high spatial resolution light field sub-aperture images. Experimental results show that our proposed method outperforms state-of-the-art light field super-resolution techniques.

Adaptive Content Frame Skipping for Wyner–Ziv-Based Light Field Image Compression

Light field (LF) imaging introduces attractive possibilities for digital imaging, such as digital focusing, post-capture changing of the focal plane or view point, and scene depth estimation, by capturing both spatial and angular information of incident light rays. However, LF image compression is still a great challenge, not only due to light field imagery requiring a large amount of storage space and a large transmission bandwidth, but also due to the complexity requirements of various applications. In this paper, we propose a novel LF adaptive content frame skipping compression solution by following a Wyner–Ziv (WZ) coding approach. In the proposed coding approach, the LF image is firstly converted into a four-dimensional LF (4D-LF) data format. To achieve good compression performance, we select an efficient scanning mechanism to generate a 4D-LF pseudo-sequence by analyzing the content of the LF image with different scanning methods. In addition, to further explore the high frame correlation of the 4D-LF pseudo-sequence, we introduce an adaptive frame skipping algorithm followed by decision tree techniques based on the LF characteristics, e.g., the depth of field and angular information. The experimental results show that the proposed WZ-LF coding solution achieves outstanding rate distortion (RD) performance while having less computational complexity. Notably, a bit rate saving of 53% is achieved compared to the standard high-efficiency video coding (HEVC) Intra codec.

Lensless light-field imaging through diffuser encoding

Microlens array-based light-field imaging has been one of the most commonly used and effective technologies to record high-dimensional optical signals for developing various potential high-performance applications in many fields. However, the use of a microlens array generally suffers from an intrinsic trade-off between the spatial and angular resolutions. In this paper, we concentrate on exploiting a diffuser to explore a novel modality for light-field imaging. We demonstrate that the diffuser can efficiently angularly couple incident light rays into a detected image without needing any lens. To characterize and analyse this phenomenon, we establish a diffuser-encoding light-field transmission model, in which four-dimensional light fields are mapped into two-dimensional images via a transmission matrix describing the light propagation through the diffuser. Correspondingly, a calibration strategy is designed to flexibly determine the transmission matrix, so that light rays can be computationally decoupled from a detected image with adjustable spatio-angular resolutions, which are unshackled from the resolution limitation of the sensor. The proof-of-concept approach indicates the possibility of using scattering media for lensless four-dimensional light-field recording and processing, not just for two- or three-dimensional imaging.

Modeling standard plenoptic camera by an equivalent camera array

Postprocessing of light fields enables us to extract more information from a scene compared to traditional cameras. Plenoptic cameras and camera arrays are two common methods for light field capture. It has been long recognized that the two devices are in some ways equivalent. Practically, both techniques have important constraints. Camera arrays are unable to provide high angular sampling, and the plenoptic camera can have a limited spatial sampling. In simulation, we can easily explore both constraints by simulating two-dimensional viewpoint images and combining them into a four-dimensional light field. We present a transformation for converting between equivalent plenoptic configurations and camera arrays when they capture pristine light fields produced in simulation. We use this approach to simulate light fields of simple scenes and validate our transformation by comparing the focus distance of a standard plenoptic camera and the equivalent camera array’s light field. We also show how some simple practical effects can be added to the pristine, synthetic light field via postprocessing and their effect on refocusing distance.

Improvements of resolution of light field imaging based on four-dimensional optical framing via a semi-transparent mirror.

A simple light field imaging system is proposed, which could improve the resolution of light field imaging and enhance the signal-to-noise ratio of the result image. In the experiment, the light field imaging system consists of a single CCD with a microlens array and a semi-transparent mirror. The Fourier slice theorem has been used to transform the four-dimensional (4D) light field information into an infinite number of two-dimensional (2D) slices. With the use of the semi-transparent mirror, the high spatial resolution image can be reconstructed on the terminal sensor. The proposed method can not only reduce the aliasing defocus degree in the imaging process but also improve the slice image resolution to meet the requirement of image definition.

Efficient calculation scheme for high pixel resolution non-hogel-based computer generated hologram from light field.

We propose a method that reduces the computation time and memory requirement in non-hogel-based hologram synthesis from light field data. The non-hogel-based technique synthesizes coherent complex field for a three-dimensional scene from its light field. Unlike conventional holographic stereogram, the non-hogel-based technique reconstructs continuous parabolic wavefront for individual three-dimensional object point by globally processing the light field. However, the global processing increases the computational load significantly, making it hard to synthesize holograms with high pixel resolution. The proposed technique reduces the computational burden by processing each two-dimensional angular frequency slice of the four-dimensional light field independently. Hologram tiling technique is also proposed to make the hologram synthesis process scalable. Using the hologram tiling and the angular-frequency-slice-based processing, 25K×25 K pixel resolution hologram was synthesized successfully.

Underwater image conversion method based on multi-layer plane refraction model with light field processing method

In order to adopt the conventional visual measurement technology to underwater application, this paper proposes an underwater image conversion method based on a multi-layer plane refraction model, which converts the underwater image into an air image. First, under the multi-layer plane refraction model, the imaging process of the underwater camera is modelled in the form of a four-dimensional light field parameterization, and the direction vector of the air image is calculated using the light field direction information. Then, the perspective projection transformation is used to obtain the corresponding pixel coordinates of the direction vector and the image plane. Finally, the transformed air image is obtained by interpolation method, using the mapping relation of the corresponding image point coordinates before and after the transformation. Simulation and experimental results show that the average error of the corrected image is 0.56 pixel, and the high-precision air image provides strong support for subsequent underwater 3D reconstruction.

Dense light field reconstruction algorithm based on dictionary learning

The camera array is an important tool to obtain the light field of target in space. The method of obtaining high angular resolution light field by a large-scaled dense camera array increases the difficulty of sampling and the equipment cost. At the same time, the demand for synchronization and transmission of a large number of data also limits the sampling rate of light field. In order to complete the dense reconstruction of sparse sampling of light field, we analyze the correlation and redundancy of multi-view images in the same scene based on sparse light field data, then establish an effective mathematical model of light field dictionary learning and sparse coding. The trained light field atoms can sparsely express the local spatial-angular consistency of light field, and the four-dimensional (4D) light field patches can be reconstructed from a two-dimensional (2D) local image patch centered around each pixel in the sensor. The global and local constraints of the four-dimensional light field are mapped into the low-dimensional space by the dictionary. These constraints are shown as the sparsity of each vector in the sparse representation domain, the constraints between the positions of non-zero elements and their values. According to the constraints among sparse encoding elements, we establish the sparse encoding recovering model of virtual angular image, and propose the sparse encoding recovering method in the transform domain. The atoms of light field in dictionary are screened and the patches of light field are represented linearly by the sparse representation matrix of the virtual angular image. In the end, the virtual angular images are constructed by image fusion after sparse inverse transform. According to multi-scene dense reconstruction experiments, the effectiveness of the proposed method is verified. The experimental results show that the proposed method can recover the occlusion, shadow and complex illumination in satisfying quality. That is to say, it can be used for dense reconstruction of sparse light field in complex scene. In our study, the dense reconstruction of linear sparse light field is achieved. In the future, the dense reconstruction of nonlinear sparse light field will be studied to promote the practical application of light field imaging.