Light Field Image Processing Research Articles

High-quality reproduction method for three-dimensional light-field displays using parallax-view information synthesis and aberration precorrection

Limited by the resolution of display panel and residual aberrations of optical components, it is difficult for the three-dimensional light-field displays (3D LFDs) to provide viewers with awesome light field reproduction. Starting from the optical process of reproducing a 3D scene by LFD, a light field display degradation model is constructed to measure the display effect observed by viewer at a specific perspective. Considering the prevalent usage of deep learning techniques in the realm of light field image processing, a novel method based on parallax-view information synthesis and aberration precorrection is proposed to elevate the display performance. This method leverages the complementary parallax information between different perspectives parallax images to enhance the image quality of single-view and introduces a convolutional neural network to estimate the optical components aberrations. A series of optimized parallax images in multiple perspectives are generated and loaded on the display panel through coding, realizing the reproduction of original 3D light field. Simulation experiments and experiments on an LFD prototype verify the effectiveness of the proposed method which can achieve higher quality 3D reproduction.

P‐2.29: Depth Estimation of Light Field Image Based on Compressed Sensing and Multi‐clue Fusion

In the current research field of light field depth estimation, the occlusion of complex scenes and a large amount of computational data are the problems that every researcher must face. For complex occlusion scenes, this paper proposes a depth estimation method based on the fusion of adaptive defocus cues and constrained angular entropy cues, which is more robust to occlusion. At the same time, the compressed sensing theory is used to compress and reconstruct the light field image to solve the problem of a large amount of data in the process of light field image acquisition, transmission, and processing. The experimental results show that the proposed method has a good overall effect in dealing with the depth estimation of the occlusion scene, and the correct depth information can be obtained. The light field image reconstructed by compressed sensing can not only obtain good depth estimation results but also reduce the amount of data effectively.

Semi-analytic Fresnel diffraction calculation with polynomial decomposition.

The numerical method based on the fast Fourier transform (FFT) is generally applied to calculate the Fresnel diffraction field, which would suffer from sampling constraints. To break this limit, in this Letter, the semi-analytic Fresnel diffraction calculation method is proposed based on polynomial decomposition. The diffraction field is computed by using properly analytic Fresnel diffraction basis functions (FDBFs) according to the application requirements. Analytic FDBF is calculated based on Legendre or Chebyshev polynomials by using the object-domain frequency division multiplexing method. The proposed method offers arbitrary sampling, high-flexibility, and high-accuracy diffraction calculation in the full Fresnel region. The computational efficiency and accuracy of the proposed method are compared with FFT-based methods. It has potential application in light field analysis, wavefront sensing, and image processing.

Image Quality Evaluation of Light Field Image Based on Macro-Pixels and Focus Stack.

Due to the complex angular-spatial structure, light field (LF) image processing faces more opportunities and challenges than ordinary image processing. The angular-spatial structure loss of LF images can be reflected from their various representations. The angular and spatial information penetrate each other, so it is necessary to extract appropriate features to analyze the angular-spatial structure loss of distorted LF images. In this paper, a LF image quality evaluation model, namely MPFS, is proposed based on the prediction of global angular-spatial distortion of macro-pixels and the evaluation of local angular-spatial quality of the focus stack. Specifically, the angular distortion of the LF image is first evaluated through the luminance and chrominance of macro-pixels. Then, we use the saliency of spatial texture structure to pool an array of predicted values of angular distortion to obtain the predicted value of global distortion. Secondly, the local angular-spatial quality of the LF image is analyzed through the principal components of the focus stack. The focalizing structure damage caused by the angular-spatial distortion is calculated using the features of corner and texture structures. Finally, the global and local angular-spatial quality evaluation models are combined to realize the evaluation of the overall quality of the LF image. Extensive comparative experiments show that the proposed method has high efficiency and precision.

Blind light field image quality assessment by analyzing angular-spatial characteristics

Light Field Image (LFI) can simultaneously record the intensity and direction information of light rays, and provide users with strong immersion experience. However, heterogeneous artifacts may be introduced during LFI processing, which results in degradation of the perceptual quality of LFI. To evaluate the LFI quality effectively, a novel blind light field image quality assessment method by analyzing angular-spatial characteristics is proposed. The main strategy is to quantify the LFI quality degradation by evaluating the angular consistency and spatial quality simultaneously. Firstly, the multi-directional intra-block and inter-block differential operations are employed on macro-pixels to generate Macro-Pixel Residual Blocks (MPRBs) on hue, saturation and luminance of LFI. Secondly, effective perceptual feature extraction schemes based on MPRBs entropy distribution and natural scene statistics of discrete cosine transform coefficients for MPRBs are developed to measure the angular consistency on each color descriptor. Thirdly, Macro-Pixel Variance (MPV) map is defined, and the quality-aware features are extracted from MPV map to measure the occlusion areas of LFI. Fourthly, the perceptual features are extracted from sub-aperture images to comprehensively measure the spatial quality of LFI. Finally, all the extracted features are pooled to predict the objective quality of LFI. Extensive experimental results on four LFI datasets show that the proposed method significantly outperforms the representative 2D, 3D, multi-view image quality assessment methods as well as the state-of-the-art LFI quality assessment methods, and is more in line with the human visual perception.

No-reference light field image quality assessment based on depth, structural and angular information

Light field imaging can capture more abundant scene information, including angular and spatial information, compared with traditional imaging technologies. However, light field image (LFI) processing will inevitably introduce distortion which is different from traditional imaging process. Therefore, LFI quality assessment has become one of important research issues on LFI processing. In this paper, we propose a depth, structural and angular information based no-reference LFI quality assessment method. Firstly, the horizontal and vertical mean difference images (MDIs) are defined to integrate the difference images of LFI's sub-aperture images (SAIs) and reflect the horizontal and vertical depth and structural information of LFI. Considering the multi-channel characteristics of human visual system, an effective feature extraction scheme with Curvelet decomposition is designed for MDIs and SAIs of the distorted LFI. As one of the representations of LFI, epipolar plane images (EPIs) contain LFI's angular and depth information, which reflect the angular consistency of LFI more intuitively. Therefore, an algorithm, namely local maximum similarity index statistics, is designed to extract the directional features from horizontal and vertical EPIs. Finally, the quality of distorted LFI is predicted by pooling these extracted features. Experimental results on three LFI datasets show that the proposed method can achieve better performance, compared with the representative traditional image quality assessment methods as well as the state-of-the-art LFI quality assessment methods.

Pseudoreference Subaperture Images and Microlens Image-Based Blind Light Field Image Quality Measurement

Light field image (LFI) records the radiation intensity and direction information of light in scene, offering powerful capabilities for computer vision. It is found that the distortion caused by LFI processing is obviously reflected in the two representations of LFI, that is, subaperture images (SAIs) and microlens images (MLI). Specifically, on the one hand, pseudoreference SAI (PSAI) of a distorted SAI, which is rendered with the adjacent distorted SAIs, can be used to measure the distortion of the involved SAIs. On the other hand, macropixel blocks (MPBs) of MLI can be adopted to measure LFI global angular distortion. Therefore, a PSAIs- and MLI-based blind LFI quality measurement (PM-BLFIQM) method is proposed in this article. First, the local light field (LLF) is defined to generate PSAI of the central distorted SAI in each LLF of LFI, and then a pseudo-full-reference quality measurement metric based on PSAIs is presented. Second, to measure the angular consistency of the distorted LFI, an MLI-based feature extraction scheme is proposed for MPBs and their mean removal version by using discrete cosine transform and singular value decomposition. Finally, the quality of the distorted LFI is predicted with the features extracted from LLFs and MLI. Experimental results demonstrate that both PSAI and MPB are effective for measuring the quality of distorted LFI. In addition, the results on three LFI databases show that the proposed method achieves higher Spearman rank-order correlation coefficient (SROCC) and Pearson linear correlation coefficient (PLCC) values, so it outperforms the state-of-the-art methods in terms of consistency with human visual perception.

High-resolution 3D light-field imaging.

.Significance: High-speed 3D imaging methods have been playing crucial roles in many biological discoveries.Aim: We present a hybrid light-field imaging system and image processing algorithm that can visualize high-speed biological events.Approach: The hybrid light-field imaging system uses the selective plane optical illumination, which simultaneously records a high-resolution 2D image and a low-resolution 4D light-field image. The high-resolution 4D light-field image is obtained by applying the hybrid algorithm derived from the deconvolution and phase retrieval methods.Results: High-resolution 3D imaging at a speed of 100-s volumes per second over an imaging field of in the , , and axis, respectively, is achieved with a 2.5 times enhancement in lateral resolution over the entire imaging field compared with standard light-field systems. In comparison to the deconvolution algorithm, the hybrid algorithm addresses the artifact issue at the focal plane and reduces the computation time by a factor of 4.Conclusions: The new hybrid light-field imaging method realizes high-resolution and ultrafast 3D imaging with a compact setup and simple algorithm, which may help discover important applications in biophotonics to visualize high-speed biological events.

Full Reference Light Field Image Quality Evaluation Based on Angular-Spatial Characteristic

The quality evaluation is an indispensable link in light field (LF) image processing. Most of existing LF objective evaluation indexes do not make effective use of the angular characteristic of LF, so the evaluation results are unsatisfactory. In this letter, the quality evaluation of LF image is constructed based on human visual system (HVS) and LF angular-spatial characteristics. Based on the fact that HVS has different sensitivity to different parallaxes, we assume that LF image quality perceived by human eyes has the optimal parallax range. A dual-fan filter is used to constrain the parallax range. Then, the overall quality of the LF is represented by combining the spatial and angular quality, which performed from the central sub-aperture image and the focus stack, respectively. In addition, because the difference of Gaussian (DoG) operator can simulate the process of extracting texture structure by human eyes. The structural similarity of DoG texture feature is utilized in the spatial quality evaluation. Extensive comparison experiments show that the proposed method is more consistent with the characteristics of LF.

Light-field image processing using deep neural network

Light-field image processing using deep neural network

Light-field image processing using deep neural network

Light-field image processing using deep neural network

Generating Stereoscopic Images With Convergence Control Ability From a Light Field Image Pair

With the advances in commercial light field cameras, light field image processing has attracted considerable attention from researchers. In this paper, we propose a novel method for generating stereoscopic images from a light field image pair with flexible control over the convergence of the virtual stereo cameras. We have developed a light field image-capturing prototype that consists of two horizontally arranged light field cameras (i.e., with their optical axes being parallel to each other). When using our proposed device for image/video capture, stereo photographers can concentrate on how to capture the desired visual experience without being frequently disturbed by having to manipulate the stereo camera parameters, i.e., the convergence angle of a stereo camera. During postprocessing, our method estimates accurate disparity maps for the light field image pair and then generates the target stereoscopic images that satisfy the desired stereo camera convergence requirements by adopting a novel view synthesis method for light field images. We have conducted extensive experiments to demonstrate the effectiveness of our proposed method.

Flame temperature estimation from light field image processing.

We propose a novel flame temperature estimation method based on a flame light field sectioned imaging model of complex temperature distribution in different media. The proposed method relies on multi-pixel reconstruction to improve the resolution of sub-aperture images. In addition, the wavelet transform denoises the flame refocused image, and then the Lucy-Richardson algorithm deconvolves the image. The temperature estimation accuracy using the proposed method is higher than that reported in a previous work, with a larger temperature estimation range from 1250K to 1800K. Moreover, we found that deconvolution plays an important role in determining the temperature estimation accuracy.

Focused-Region Segmentation for Refocusing Images from Light Fields

Since the focused regions of the light-field refocusing image contain depth cues, focused-region segmentation becomes a fundamental step of depth estimation, 3D measurement, and visual measurement. However, in the emerging field of light-field image processing, recent research has emphasized focus evaluation rather than a systematic method for focused-region segmentation. The segmentation algorithms for low depth-of-field images are of significance for this problem, but those algorithms have high time complexity and are not suitable for the computationally intensive applications of light-field imaging. Therefore, based on the pulse synchronous mechanism of the pulse coupled neural network (PCNN), we establish the model of neural firing sequence and some criteria of pixel classification. Further, we design an algorithm of the focused-region segmentation and its parameter settings. The experimental results show that the proposed method segments the refocusing images faster than alternative methods and meets the needs of light-field image processing and related applications.

Introduction to the Issue on Light Field Image Processing

Introduction to the Issue on Light Field Image Processing

Light Field Image Processing: An Overview

Light field imaging has emerged as a technology allowing to capture richer visual information from our world. As opposed to traditional photography, which captures a 2D projection of the light in the scene integrating the angular domain, light fields collect radiance from rays in all directions, demultiplexing the angular information lost in conventional photography. On the one hand, this higher dimensional representation of visual data offers powerful capabilities for scene understanding, and substantially improves the performance of traditional computer vision problems such as depth sensing, post-capture refocusing, segmentation, video stabilization, material classification, etc. On the other hand, the high-dimensionality of light fields also brings up new challenges in terms of data capture, data compression, content editing, and display. Taking these two elements together, research in light field image processing has become increasingly popular in the computer vision, computer graphics, and signal processing communities. In this paper, we present a comprehensive overview and discussion of research in this field over the past 20 years. We focus on all aspects of light field image processing, including basic light field representation and theory, acquisition, super-resolution, depth estimation, compression, editing, processing algorithms for light field display, and computer vision applications of light field data.

Special Issue on Light Field Image Processing

Special Issue on Light Field Image Processing

Special Issue on Light Field Image Processing

Special Issue on Light Field Image Processing

Depth and all-in-focus imaging by a multi-line-scan light-field camera

We present a multi-line-scan light-field image acquisition and processing system designed for 2.5/3-D inspection of fine surface structures in industrial environments. The acquired three-dimensional light field is composed of multiple observations of an object viewed from different angles. The acquisition system consists of an area-scan camera that allows for a small number of sensor lines to be extracted at high frame rates, and a mechanism for transporting an inspected object at a constant speed and direction. During acquisition, an object is moved orthogonally to the camera’s optical axis as well as the orientation of the sensor lines and a predefined subset of lines is read out from the sensor at each time step. This allows for the construction of so-called epipolar plane images (EPIs) and subsequent EPI-based depth estimation. We compare several approaches based on testing a set of slope hypotheses in the EPI domain. Hypotheses are derived from block matching, namely the sum of absolute differences, modified sum of absolute differences, normalized cross correlation, census transform, and modified census transform. Results for depth estimation and all-in-focus image generation are presented for synthetic and real data.