Plenoptic Image Research Articles

Design of a super-resolution scheme for plenoptic images

Design of a super-resolution scheme for plenoptic images

Assessing the 3D resolution of refocused correlation plenoptic images using a general-purpose image quality estimator

Correlation plenoptic imaging (CPI) is emerging as a promising approach to light-field imaging (LFI), a technique enabling simultaneous measurement of light intensity distribution and propagation direction from a scene. LFI allows single-shot 3D sampling, offering fast 3D reconstruction for a wide range of applications. However, the array of micro-lenses typically used in LFI to obtain 3D information limits image resolution, which rapidly declines with enhanced volumetric reconstruction capabilities. CPI addresses this limitation by decoupling the measurement of the light field on two photodetectors with spatial resolution, eliminating the need for micro-lenses. 3D information is encoded in a four-dimensional correlation function, which is decoded in post-processing to reconstruct images without the resolution loss seen in conventional LFI. This paper evaluates the tomographic performance of CPI, demonstrating that the refocusing reconstruction method provides axial sectioning capabilities comparable to conventional imaging systems. A general-purpose analytical approach based on image fidelity is proposed to quantitatively study axial and lateral resolution. The analysis fully characterizes the volumetric resolution of any CPI architecture, offering a comprehensive evaluation of its imaging performance.

Development of raw-image ray-bundling for focused and unfocused plenoptic 3D-PTV

This paper introduces a new plenoptic 3D particle tracking velocimetry (3D-PTV) technique called raw-image ray-bundling (RIRB), which combines the 3D clustering algorithm of light-field ray-bundling (LFRB) with the raw image segmentation concept of epipolar triangular connectivity (ETC). The combined approach provides flexibility towards accommodating both unfocused and focused plenoptic cameras. Additionally, the RIRB algorithm bypasses the computational step of perspective-view generation in LFRB, thereby allowing lower specification microlens arrays (MLA) to be used without excessive loss of measurement resolution. As part of RIRB’s development, synthetic particle-field data and experimental dot-panel targets were used to demonstrate the viability of RIRB. The latter was performed through a VEO640 high-speed camera retrofitted for plenoptic imaging with a 2.5 mm focal-length, F/10 MLA. For the synthetic dataset, RIRB demonstrated better performance than LFRB in processing low-resolution images, with mean errors below 0.85% of the field-of-view (FOV). Additionally, a synthetic Hagen-Poiseuille flow dataset was used to demonstrate the feasibility of RIRB for 3D flow velocimetry using both high and low-resolution images. For the experimental dataset, x/y errors below 0.15% of the FOV and z error below 1.0% were achieved. The combination of synthetic and experimental validations suggests RIRB is a promising approach for 3D triangulation from both focused and unfocused raw plenoptic images, overcoming the low-resolution issue inherent with perspective-view decoding in plenoptic systems.

Improving depth uncertainty in plenoptic camera-based velocimetry

This work describes the development of a particle tracking velocimetry (PTV) algorithm designed to improve three-dimensional (3D), three-component velocity field measurements using a single plenoptic camera. Particular focus is on mitigating the longstanding depth uncertainty issues that have traditionally plagued plenoptic particle image velocimetry (PIV) experiments by leveraging the camera’s ability to generate multiple perspective views of a scene in order to assist both particle triangulation and tracking. 3D positions are first estimated via light field ray bundling (LFRB) whereby particle rays are projected into the measurement volume using image-to-object space mapping. Tracking is subsequently performed independently within each perspective view, providing a statistical amalgamation of each particle’s predicted motion through time in order to help guide 3D trajectory estimation while simultaneously protecting the tracking algorithm from physically unreasonable fluctuations in particle depth positions. A synthetic performance assessment revealed a reduction in the average depth errors obtained by LFRB as compared to the conventional multiplicative algebraic reconstruction technique when estimating particle locations. Further analysis using a synthetic vortex ring at a magnification of − 0.6 demonstrated plenoptic-PIV capable of maintaining the equivalent of 0.1–0.15 voxel accuracy in the depth domain at a spacing to displacement ratio of 5.3–10.5, an improvement of 84–89% compared to plenoptic-PIV. Experiments were conducted at a spacing to displacement ratio of approximately 5.8 to capture the 3D flow field around a rotor within the rotating reference frame. The resulting plenoptic-PIV/PTV vector fields were evaluated with reference to a fixed frame stereoscopic-PIV (stereo-PIV) validation experiment. A systematic depth-wise (radial) component of velocity directed toward the wingtip, consistent with observations from prior literature and stereo-PIV experiments, was captured by plenoptic-PTV at magnitudes similar to the validation data. In contrast, the plenoptic-PIV did not discern any coherent indication of radial motion. Our algorithm constitutes a significant advancement in enhancing the functionality and versatility of single-plenoptic camera flow diagnostics by directly addressing the primary limitation associated with plenoptic imaging.Graphical abstract

Learned Focused Plenoptic Image Compression With Microimage Preprocessing and Global Attention

Focused plenoptic cameras can record spatial and angular information of the light field (LF) simultaneously with higher spatial resolution relative to traditional plenoptic cameras, which facilitate various applications in computer vision. However, the existing plenoptic image compression methods present ineffectiveness to the captured images due to the complex micro-textures generated by the microlens relay imaging and long-distance correlations among the microimages. In this paper, a lossy end-to-end learning architecture is proposed to compress the focused plenoptic images efficiently. First, a data preprocessing scheme is designed according to the imaging principle to remove the sub-aperture image ineffective pixels in the recorded light field and align the microimages to the rectangular grid. Then, the global attention module with large receptive field is proposed to capture the global correlation among the feature maps using pixel-wise vector attention computed in the resampling process. Also, a new image dataset consisting of 1910 focused plenoptic images with content and depth diversity is built to benefit training and testing. Extensive experimental evaluations demonstrate the effectiveness of the proposed approach. It outperforms intra coding of HEVC and VVC by an average of 62.57% and 51.67% bitrate reduction on the 20 preprocessed focused plenoptic images, respectively. Also, it achieves 18.73% bitrate saving and generates perceptually pleasant reconstructions compared to the state-of-the-art end-to-end image compression methods, which benefits the applications of focused plenoptic cameras greatly. The dataset and code are publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/VincentChandelier/GACN</uri> .

Deep learning approach for denoising low-SNR correlation plenoptic images

Correlation Plenoptic Imaging (CPI) is a novel volumetric imaging technique that uses two sensors and the spatio-temporal correlations of light to detect both the spatial distribution and the direction of light. This novel approach to plenoptic imaging enables refocusing and 3D imaging with significant enhancement of both resolution and depth of field. However, CPI is generally slower than conventional approaches due to the need to acquire sufficient statistics for measuring correlations with an acceptable signal-to-noise ratio (SNR). We address this issue by implementing a Deep Learning application to improve image quality with undersampled frame statistics. We employ a set of experimental images reconstructed by a standard CPI architecture, at three different sampling ratios, and use it to feed a CNN model pre-trained through the transfer learning paradigm U-Net architecture with VGG-19 net for the encoding part. We find that our model reaches a Structural Similarity (SSIM) index value close to 1 both for the test sample (SSIM = 0.87 pm 0.02) and in 5-fold cross validation (SSIM = 0.92 pm 0.07); the results are also shown to outperform classic denoising methods, in particular for images with lower SNR. The proposed work represents the first application of Artificial Intelligence in the field of CPI and demonstrates its high potential: speeding-up the acquisition by a factor 20 over the fastest CPI so far demonstrated, enabling recording potentially 200 volumetric images per second. The presented results open the way to scanning-free real-time volumetric imaging at video rate, which is expected to achieve a substantial influence in various applications scenarios, from monitoring neuronal activity to machine vision and security.

Learning to detect 3D planes from a single plenoptic image.

In this paper, we propose a deep neural network to detect 3D planar surfaces from single plenoptic images captured by a Lytro Illum camera. Different from learning methods based on a single RGB image, we train a network to exploit the light distribution information both in spatial and angular dimensions from multi-sub-aperture images. The features from each sub-aperture image are extracted by using parameter-sharing convolutional layers and then fused to jointly infer the parameters of planes, depths, and segmentation masks. The experiments demonstrate that our approach outperforms the existing state-of-the-art methods with significant margins in the plane detection metrics.

Direct wavefront sensing with a plenoptic sensor based on deep learning.

Traditional plenoptic wavefront sensors (PWS) suffer from the obvious step change of the slope response which leads to the poor performance of phase retrieval. In this paper, a neural network model combining the transformer architecture with the U-Net model is utilized to restore wavefront directly from the plenoptic image of PWS. The simulation results show that the averaged root mean square error (RMSE) of residual wavefront is less than 1/14λ (Marechal criterion), proving the proposed method successfully breaks through the non-linear problem existed in PWS wavefront sensing. In addition, our model performs better than the recently developed deep learning models and traditional modal approach. Furthermore, the robustness of our model to turbulence strength and signal level is also tested, proving the good generalizability of our model. To the best of our knowledge, it is the first time to perform direct wavefront detection with a deep-learning-based method in PWS-based applications and achieve the state-of-the-art performance.

Transient leading-edge vortex development on a wing rolling in uniform flow

Plenoptic particle image velocimetry and surface pressure measurements were used to analyse the early development of leading-edge vortices (LEVs) created by a flat-plate wing of aspect ratio 2 rolling in a uniform flow parallel to the roll axis. Four cases were constructed by considering two advance coefficients, $J=0.54$ and 1.36, and two wing radii of gyration, $R_g/c=2.5$ and 3.25. In each case, the wing pitch angle was articulated such as to achieve an angle of attack of $33^{\circ }$ at the radius of gyration of the wing. The sources and sinks of vorticity were quantified for a chordwise rectangular control region, using a vorticity transport framework in a non-inertial coordinate system attached to the wing. Within this framework, terms associated with Coriolis acceleration provide a correction to tilting and spanwise convective fluxes measured in the rotating frame and, for the present case, have insignificant values. For the baseline case ( $J=0.54, R_g/c=3.25$ ), three distinct spanwise regions were observed within the LEV, with distinct patterns of vortex evolution and vorticity transport mechanisms in each region. Reducing the radius of gyration to $R_g/c=2.5$ resulted in a more stable vortex with the inboard region extending over a broader spanwise range. Increasing advance ratio eliminated the conical vortex, resulting in transport processes resembling the mid-span region of the baseline case. Although the circulation of the LEV system was generally stronger at the larger advance coefficient, the shear-layer contribution was diminished.

Ray-Space Motion Compensation for Lenslet Plenoptic Video Coding.

Plenoptic images and videos bearing rich information demand a tremendous amount of data storage and high transmission cost. While there has been much study on plenoptic image coding, investigations into plenoptic video coding have been very limited. We investigate the motion compensation (or so-called temporal prediction) for plenoptic video coding from a slightly different perspective by looking at the problem in the ray-space domain instead of in the conventional pixel domain. Here, we develop a novel motion compensation scheme for lenslet video under two sub-cases of ray-space motion, that is, integer ray-space motion and fractional ray-space motion. The proposed new scheme of light field motion-compensated prediction is designed such that it can be easily integrated into well-known video coding techniques such as HEVC. Experimental results compared to relevant existing methods have shown remarkable compression efficiency with an average gain of 20.03% and 21.76% respectively under "Low delayed B " and "Random Access" configurations of HEVC.

Tensor-based plenoptic image denoising by integrating super-resolution

In this paper, we propose a novel tensor-based denoising method targeting at plenoptic images which contain 4D light field (2D angular + 2D spatial) and 5D hyperspectral light field (2D angular + 2D spatial + 1D spectral). In order to make use of the high-dimension structural property of plenoptic images, we first generalize the intrinsic tensor sparsity measure to plenoptic images by extending the nonlocal similarity from the spatial dimension to the angular dimension. Second, to eliminate the sub-pixel misalignment of different views, we integrate the spatial super-resolution into denoising and exploit the spatial-angular correlation by utilizing the nonlocal similarity of the refined high-resolution central view. In the procedure of super-resolution, we utilize an intensity consistency criterion and a coordinate rationality criterion to facilitate the process of projection. The denoising performance can be boosted after back-projection performed on the refined high-resolution central view. Experimental results validate the superior performance of the proposed method on several plenoptic image datasets in terms of both subjective and objective quality.

A First Attempt to Combine NIRS and Plenoptic Cameras for the Assessment of Grasslands Functional Diversity and Species Composition

Grassland represents more than half of the agricultural land. Numerous metrics (biomass, functional trait, species composition) can be used to describe grassland vegetation and its multiple functions. The measures of these metrics are generally destructive and laborious. Indirect measurements using optical tools are a possible alternative. Some tools have high spatial resolutions (digital camera), and others have high spectral resolutions (Near Infrared Spectrometry NIRS). A plenoptic camera is a multifocal camera that produces clear images at different depths in an image. The objective of this study was to test the interest of combining plenoptic images and NIRS data to characterize different descriptors of two Mediterranean legumes mixtures. On these mixtures, we measured biomass, species biomass, and functional trait diversity. NIRS and plenoptic images were acquired just before the field measurements. The plenoptic images were analyzed using Trainable Weka Segmentation ImageJ to evaluate the percentage of each species in the image. We calculated the average and standard deviation of the different colors (red, green, blue reflectance) in the image. We assessed the percentage of explanation of outputs of the images and NIRS analyses using variance partition and partial least squares. The biomass Trifolium michelianum and Vicia sativa were predicted with more than 50% variability explained. For the other descriptors, the variability explained was lower but nevertheless significant. The percentage variance explained was nevertheless quite low, and further work is required to produce a useable tool, but this work already demonstrates the interest in combining image analysis and NIRS.

Volumetric Velocity Measurements of a Three-Dimensional Shock-Wave/Boundary-Layer Interaction with Flow Actuation Using Two-Camera Plenoptic Particle Image Velocimetry

Two-camera plenoptic particle image velocimetry is used to study a three-dimensional (3-D) shock-wave/boundary-layer interaction (SBLI) generated by a single unswept fin. The SBLI considered was the swept interaction induced by a sharp fin at an angle of attack of 15 deg in Mach 2 flow. The fin generates an oblique shock with a wave angle of approximately 45 deg that interacts with the boundary layer on the incident flat plate. As an extension of previous work, these experiments were conducted to examine the effects of momentum-based flow actuation on the 3-D flowfield, as well as reduce the data uncertainty and improve the overall data quality via addition of a second camera. The following three cases are presented for this work: 1) the baseline interaction with the presence of the inactive microjet actuators (physical insert present), 2) the mean SBLI flowfield under the influence of microjet actuation upstream of the oblique shock, and 3) the mean flowfields corresponding to phase-locked images captured at four different phases of the actuator pressure cavity signal. The results show good correlation with the literature of the SBLI with actuation, including a possible shift or rotation of the shock feature during actuation, and an increased velocity deficit and boundary-layer height. The resonance-enhanced microjet actuation also had a notable impact on the low-speed flow region near the fin. This work presents one of the first 3-D measurements of a fin-generated SBLI influenced by flow actuation.

REPAID: resolution-enhanced plenoptic all-in-focus imaging using deep neural networks.

Due to limited depth-of-focus, classical 2D images inevitably lose details of targets out of depth-of-focus, while all-in-focus images break through the limit by fusing multi-focus images, thus being able to focus on targets in extended depth-of-view. However, conventional methods can hardly obtain dynamic all-in-focus imaging in both high spatial and temporal resolutions. To solve this problem, we design REPAID, meaning resolution-enhanced plenoptic all-in-focus imaging using deep neural networks. In REPAID, multi-focus images are first reconstructed from a single-shot plenoptic image, then upsampled using specially designed deep neural networks suitable for real scenes without ground truth to finally generate all-in-focus image in both high temporal and spatial resolutions. Experiments on both static and dynamic scenes have proved that REPAID can obtain high-quality all-in-focus imaging when using simple setups only; therefore, it is a promising tool in applications especially intended for imaging dynamic targets in large depth-of-view.

A robust Plenoptic image watermarking method using graph-based transform

Numerous methods have been introduced for digital images watermarking. But there are fewer studies on plenoptic images. This image has more information (such as depth) and costs more than regular digital images. So, protecting the ownership of plenoptic is important. In this paper a robust plenoptic image watermarking has been introduced which is based on Graph-based Transform (GBT). Also, in order to maximizing the robustness of watermarking an optimization has been used to find the best graph structure of GBT. This optimization has done using a Genetic Algorithm. One of the common ways to robust the watermarking is the use of discrete cosine transform (DCT). In this study, we have shown that the proposed method is much more powerful than DCT. The proposed method is tested on three plenoptic images namely Alley, Basic Shapes and Coffee Table and the results show that the proposed method is more robust to similar methods such as DCT methods. Watermarking was evaluated using BER, SSIM and PSNR criteria and Gaussian noise attacks with varying intensities.

Lossless Compression of Plenoptic Camera Sensor Images

We propose a codec for the lossless compression of plenoptic camera sensor images. The proposed encoder starts by splitting the input lenslet image into rectangular patches, with each patch corresponding to a microlens image. The encoder and decoder exploit the correlation between the pixels in neighbor patches using a patch-by-patch prediction mechanism where each pixel of a patch has its own dedicated sparse predictor designed to utilize the most relevant pixels from the neighbor patch to the left. An intra-patch prediction mask together with the pixels from the neighbor left patch form the final prediction template. The encoder performs the design of sparse predictors by first finding the relevant regressors in the final template. The patches are classified into M classes according to two possible mechanisms (either based on depth information or based on Bayer mask colors), and the sparse predictor design is performed for each pair (class label; patch pixel index). A relevant context selection mirrors the selection of relevant regressors, thereby providing the arithmetic coding with skewed coding distributions at each context. We show examples of the application of the proposed methods on sensor images from the JPEG Pleno database, thus demonstrating the improved performance of the proposed methods compared to the existing predictive methods for encoding camera sensor images.

Optimal Design of Microlens Aperture Size for Focused Plenoptic Camera

The aperture size of microlens is a key parameter of a plenoptic camera, which strongly influence the performance of the camera. Generally, the size of microlens aperture is submillimeter, which will introduce obvious diffraction effect with large f-number. Besides, optical distortions of plenoptic camera also bring degradation effect in raw plenoptic image. In this letter, based on wave optics, a more accurate imaging model with diffraction effect and optical distortion for focused plenoptic camera is proposed. The modulation transfer function (MTF) evaluation suggests that the proposed model is more close to the real one when compared to the previous models. Additionally, the raw images under different microlens aperture sizes are synthesized. Finally, a relatively optimal size of microlens aperture is obtained according the results of depth estimation.

Single-Camera Three-Dimensional Velocity Measurement of a Fin-Generated Shock-Wave/Boundary-Layer Interaction

The three-dimensional (3-D) shock-wave/boundary-layer interaction (SBLI) between a fin-generated shock and a turbulent boundary layer is examined using plenoptic particle image velocimetry (PIV): a 3-D diagnostic technique that uses a single camera to obtain direct 3-D velocity measurements. A sharp, unswept fin at a 15 deg angle of attack in Mach 2 flow generates a 45 deg oblique shock that interacts with the boundary layer on the tunnel floor. This type of SBLI has been extensively studied in quasi-two-dimensional representations, but no work up to this point has been done to capture true 3-D measurements. This work represents the first 3-D velocity field measurements in supersonic flow using plenoptic PIV, as well as the first 3-D velocity measurements of a fin-generated SBLI. Flow features including the -shock structure and slip region expected from this SBLI were able to be resolved, and the 3-D imaging allowed for better visualization of the overall flow topology. Plenoptic PIV enabled the visualization of features not considered in previous literature, particularly the 3-D curvature of the inviscid shock and effects from an expansion region emanating from the fin tip due to the finite fin height. Plenoptic PIV results are compared with inviscid shock theory and stereoscopic PIV for validation purposes.

High efficiency dense light field and all-in-focus compression for lossless satellite image by using CCSDS

Light-field enables us to observe scenes from free view-points. However, it generally consists of 4-D enormous data, which are not suitable for storing or transmitting without effective compression. Image acquisition, transmission, interpretation and visualization have witnessed a lot of research interest and have manifold applications in real life. Light field Images popularly known as Plenoptic Images are light rays travelling in every direction. The data sets generated using plenoptic images are large in size and consume a lot of storage space. This paper studies the general process of image acquisition, light field images and various datasets. Image compression will help the users or professionals to not only achieve storage efficiency but also aid in transmission of Images. In this paper, we study efficient compression of multi-focus images synthesized from dense light-field by using DWT instead of DCT-based compression in order to suppress degradation such as block noise. Quality of reconstructed light-field is evaluated by PSNR for analyzing characteristics of residuals. This paper presents the modeling, design and implementation of two IP cores that are compliant with the CCSDS 121.0-B-2 and CCSDS 123.0-B-1 lossless satellite image compression standards. This paper provides a comparative study of various image compression techniques/algorithms and evaluates them on parameters like peak signal to noise ratio (PSNR), mean square error (MSE), Structural Similarity Index (SSI) and compression ratio (CR).

Radiometric model for plenoptic image formation

This paper presents a general plenoptic image formation model based on light ray radiometry. Most existing plenoptic image formation models focus on projection geometry inside a camera equipped with a specific optical device, such as an array of micro-thin-lenses, while light ray radiometry is incomplete or simply neglected. This study aims to describe a general view of light ray radiometry that is then used to define a plenoptic image formation model. This model is based on a radiometry model and light ray projection geometry; hence, it explains phenomena, such as light transport and light direction. Moreover, the proposed plenoptic image formation model describes several radiometric phenomena, such as haze, defocus blur or natural vignetting. The genericity of this model is shown by retrieving existing non-plenoptic models from it. Moreover, the proposed model is assessed by simulating various phenomena, including vignetting and defocus blur, from both synthetic and real data.