Light Field Imaging Research Articles

Visual Security Index Combining CNN and Filter for Perceptually Encrypted Light Field Images

Visual security index (VSI) represents a quantitative index for the visual security evaluation of perceptually encrypted images. Recently, the research on visual security of encrypted light field (LF) images faces two challenges. One is that the existing perceptually encrypted image databases are often too small, which is easy to cause overfitting in convolutional neural network (CNN). The other is that existing VSI models did not take a full account the intrinsic characteristics of the LF images and highly relied on handcrafted feature extraction. In this article, we construct a new database of perceptually encrypted LF images, called the PE-SLF, which is 2.6 times as big as the existing largest perceptual encrypted image database. Moreover, a novel visual security index (VSI) model is proposed by taking into full consideration the intrinsic spatial-angular characteristics of the LF images and the outstanding capabilities of CNN in feature extraction. First, we exploit CNN to detect the texture and structure features of encrypted sub-aperture images in the spatial domain. Second, we apply the Gabor filter to detect the Gabor feature over the epi-polar plane images in angular domain. Last, the spatial and angular similarity measurements are subsequently calculated for jointly yielding the final visual security score. Experimental results on the constructed PE-SLF demonstrate that the proposed VSI model is closer to the perception of HVS in visual security evaluation of encrypted LF images compared to other classical and state-of-the-art models.

Lightfield hyperspectral imaging in neuro-oncology surgery: an IDEAL 0 and 1 study.

Hyperspectral imaging (HSI) has shown promise in the field of intra-operative imaging and tissue differentiation as it carries the capability to provide real-time information invisible to the naked eye whilst remaining label free. Previous iterations of intra-operative HSI systems have shown limitations, either due to carrying a large footprint limiting ease of use within the confines of a neurosurgical theater environment, having a slow image acquisition time, or by compromising spatial/spectral resolution in favor of improvements to the surgical workflow. Lightfield hyperspectral imaging is a novel technique that has the potential to facilitate video rate image acquisition whilst maintaining a high spectral resolution. Our pre-clinical and first-in-human studies (IDEAL 0 and 1, respectively) demonstrate the necessary steps leading to the first in-vivo use of a real-time lightfield hyperspectral system in neuro-oncology surgery. A lightfield hyperspectral camera (Cubert Ultris ×50) was integrated in a bespoke imaging system setup so that it could be safely adopted into the open neurosurgical workflow whilst maintaining sterility. Our system allowed the surgeon to capture in-vivo hyperspectral data (155 bands, 350-1,000 nm) at 1.5 Hz. Following successful implementation in a pre-clinical setup (IDEAL 0), our system was evaluated during brain tumor surgery in a single patient to remove a posterior fossa meningioma (IDEAL 1). Feedback from the theater team was analyzed and incorporated in a follow-up design aimed at implementing an IDEAL 2a study. Focusing on our IDEAL 1 study results, hyperspectral information was acquired from the cerebellum and associated meningioma with minimal disruption to the neurosurgical workflow. To the best of our knowledge, this is the first demonstration of HSI acquisition with 100+ spectral bands at a frame rate over 1Hz in surgery. This work demonstrated that a lightfield hyperspectral imaging system not only meets the design criteria and specifications outlined in an IDEAL-0 (pre-clinical) study, but also that it can translate into clinical practice as illustrated by a successful first in human study (IDEAL 1). This opens doors for further development and optimisation, given the increasing evidence that hyperspectral imaging can provide live, wide-field, and label-free intra-operative imaging and tissue differentiation.

Achromatic and Resolution Enhancement Light Field Deep Neural Network for ZnO Nematic Liquid Crystal Microlens Array

Nematic liquid‐crystal microlens arrays (LC‐MLAs) often exhibit chromatic aberration and low resolution, severely compromising their optical imaging quality. This study proposes an achromatic and resolution enhancement light field (ARELF) deep neural network (DNN) to address these issues. The training set is constructed by incorporating LC‐MLA characteristics’ degradation, retrofitting the vimeo90k dataset. The network's hidden layer is trained to learn about chromatic aberration and low resolution of LC‐MLA while extracting imaging features and fusing the information of complementary features of a light field under varying voltages. The loss function includes both chromatic aberration and overall resolution. The light field images of ZnO LC‐MLA under seven consecutive voltages are used as input to test the proposed DNN model. After experimental verification, the proposed model effectively eliminates chromatic aberration while enhancing the spatial and temporal resolution of LC‐MLA. This novel network can be utilized to optimize the design process of LC‐MLA and significantly improve its imaging performance.

No-Reference Light Field Image Quality Assessment Exploiting Saliency

In the near future, the broadcasting scenario will be characterized by immersive content. One of the systems for capturing the 3D content of a scene is the Light Field imaging. The huge amount of data and the specific transmission scenario impose strong constraints on services and applications. Among others, the evaluation of the quality of the received media cannot rely on the original signal but should be based only on the received data. In this direction, we propose a no-reference quality metric for light field images which is based on spatial and angular characteristics. In more details, the estimated saliency and cyclopean maps of light field images are exploited to extract the spatial features. The angular consistency features are, instead, measured with the use of the Global Luminance Distribution knowledge and the Weighted Local Binary Patterns operator on Epipolar Plane Images. The effectiveness of the proposed metric is assessed by comparing its performance with state-of-the-art quality metrics using 4 datasets: SMART, Win5-LID, VALID 10-bit, and VALID 8-bit. Furthermore, the performance is analyzed in cross-datasets, with different distortions, and for different saliency maps. The achieved results show that the performance of the proposed model outperforms state-of-the-art approaches and perform well for different distortion types and with various saliency models.

Disparity-Guided Light Field Image Super-Resolution via Feature Modulation and Recalibration

Disparity-Guided Light Field Image Super-Resolution via Feature Modulation and Recalibration

FPattNet: A Multi-Scale Feature Fusion Network with Occlusion Awareness for Depth Estimation of Light Field Images.

A light field camera can capture light information from various directions within a scene, allowing for the reconstruction of the scene. The light field image inherently contains the depth information of the scene, and depth estimations of light field images have become a popular research topic. This paper proposes a depth estimation network of light field images with occlusion awareness. Since light field images contain many views from different viewpoints, identifying the combinations that contribute the most to the depth estimation of the center view is critical to improving the depth estimation accuracy. Current methods typically rely on a fixed set of views, such as vertical, horizontal, and diagonal, which may not be optimal for all scenes. To address this limitation, we propose a novel approach that considers all available views during depth estimation while leveraging an attention mechanism to assign weights to each view dynamically. By inputting all views into the network and employing the attention mechanism, we enable the model to adaptively determine the most informative views for each scene, thus achieving more accurate depth estimation. Furthermore, we introduce a multi-scale feature fusion strategy that amalgamates contextual information and expands the receptive field to enhance the network's performance in handling challenging scenarios, such as textureless and occluded regions.

Deep focus light-field camera for handheld 3D intraoral scanning using crosstalk-free solid immersion microlens arrays.

3D in vivo imaging techniques facilitate disease tracking and treatment, but bulky configurations and motion artifacts limit practical clinical applications. Compact light-field cameras with microlens arrays offer a feasible option for rapid volumetric imaging, yet their utilization in clinical practice necessitates an increased depth-of-field for handheld operation. Here, we report deep focus light-field camera (DF-LFC) with crosstalk-free solid immersion microlens arrays (siMLAs), allowing large depth-of-field and high-resolution imaging for handheld 3D intraoral scanning. The siMLAs consist of thin PDMS-coated microlens arrays and a metal-insulator-metal absorber to extend the focal length with low optical crosstalk and specular reflection. The experimental results show that the immersion of MLAs in PDMS increases the focal length by a factor of 2.7 and the transmittance by 5.6%-27%. Unlike conventional MLAs, the siMLAs exhibit exceptionally high f-numbers up to f/6, resulting in a large depth-of-field for light-field imaging. The siMLAs were fully integrated into an intraoral scanner to reconstruct a 3D dental phantom with a distance measurement error of 82 ± 41 μm during handheld operation. The DF-LFC offers a new direction not only for digital dental impressions with high accuracy, simplified workflow, reduced waste, and digital compatibility but also for assorted clinical endoscopy and microscopy.

Portrait stylized rendering for 3D light-field display based on radiation field and example guide.

With the development of three-dimensional (3D) light-field display technology, 3D scenes with correct location information and depth information can be perceived without wearing any external device. Only 2D stylized portrait images can be generated with traditional portrait stylization methods and it is difficult to produce high-quality stylized portrait content for 3D light-field displays. 3D light-field displays require the generation of content with accurate depth and spatial information, which is not achievable with 2D images alone. New and innovative portrait stylization techniques methods should be presented to meet the requirements of 3D light-field displays. A portrait stylization method for 3D light-field displays is proposed, which maintain the consistency of dense views in light-field display when the 3D stylized portrait is generated. Example-based portrait stylization method is used to migrate the designated style image to the portrait image, which can prevent the loss of contour information in 3D light-field portraits. To minimize the diversity in color information and further constrain the contour details of portraits, the Laplacian loss function is introduced in the pre-trained deep learning model. The three-dimensional representation of the stylized portrait scene is reconstructed, and the stylized 3D light field image of the portrait is generated the mask guide based light-field coding method. Experimental results demonstrate the effectiveness of the proposed method, which can use the real portrait photos to generate high quality 3D light-field portrait content.

Analysis of the relationship between display depth and 3D image definition in light-field display from visual perspective

An analysis method for the relationship between light-field display (LFD) depth and 3D image definition from visual perspective is presented in this paper. A simplified analytical model suitable for standard LFD system is established, which can be used to analyze the formation of voxels from a perspective that is more consistent with visual perception. Moreover, the formula for calculating voxel size is derived in relation to the depth plane and viewing position. Reasonable simulation programs and LFD devices with different parameters are used in experiments to verify the correctness of the theoretical analysis. The analysis and experimental results presented in this paper may contribute to a better understanding of the principle of light-field display, and it is helpful to design and optimize the LFD system with better display effect.

High-efficiency broadband achromatic metalenses for visible full-stokes polarization imaging.

Polarization-imaging technology has important applications in target detection, communication, biomedicine, and other fields. A polarization imaging system based on metalenses, which provides new possibilities for the realization of highly integrated full-Stokes polarization imaging systems, can solve the problems of traditional polarization imaging systems, such as complex structures, large volumes, and the inability to simultaneously obtain linear and circular polarization states. However, currently designed metalens arrays that can achieve real-time full-Stokes polarization imaging can generally only be used for monochromatic detection, which significantly limits the amount of measured information of the object. Broad-spectrum polarization color imaging allows more image degrees of freedom, enabling more accurate characterization of polarization for multi-target object scenes in complex environments. To achieve broad-spectrum polarization imaging, we propose and design a metalens array that can achieve full-Stokes polarization imaging in the broadband visible range, in which the design process of metalenses for splitting and focusing broadband orthogonal circularly polarized light is emphasized. To design metalenses that can achieve polarization splitting and efficient focusing, we simulate and optimize the height and period of the nano-units and show that smaller periods and larger heights do not always result in higher-performance devices when designing multifunctional metalenses. The designed metalens array can split and diffraction-limited focus the orthogonal polarized incident light to the designated position with average focusing efficiencies of 59.2% under 460-680 nm TM linearly polarized light, 53.1% under TE linearly polarized light, 58.8% under left-handed circularly polarized light, and 52.7% under right-handed circularly polarized light. The designed metalenses can be applied to imaging systems, such as polarization imaging and polarization light-field imaging systems.

Correlated-photon imaging at 10 volumetric images per second

The correlation properties of light provide an outstanding tool to overcome the limitations of traditional imaging techniques. A relevant case is represented by correlation plenoptic imaging (CPI), a quantum-inspired volumetric imaging protocol employing spatio-temporally correlated photons from either entangled or chaotic sources to address the main limitations of conventional light-field imaging, namely, the poor spatial resolution and the reduced change of perspective for 3D imaging. However, the application potential of high-resolution imaging modalities relying on photon correlations is limited, in practice, by the need to collect a large number of frames. This creates a gap, unacceptable for many relevant tasks, between the time performance of correlated-light imaging and that of traditional imaging methods. In this article, we address this issue by exploiting the photon number correlations intrinsic in chaotic light, combined with a cutting-edge ultrafast sensor made of a large array of single-photon avalanche diodes (SPADs). This combination of source and sensor is embedded within a novel single-lens CPI scheme enabling to acquire 10 volumetric images per second. Our results place correlated-photon imaging at a competitive edge and prove its potential in practical applications.

AIF-LFNet: All-in-Focus Light Field Super-Resolution Method Considering the Depth-Varying Defocus

As an aperture-divided computational imaging system, microlens array (MLA) -based light field (LF) imaging is playing an increasingly important role in computer vision. As the trade-off between the spatial and angular resolutions, deep learning (DL) -based image super-resolution (SR) methods have been applied to enhance the spatial resolution. However, in existing DL-based methods, the depth-varying defocus is not considered both in dataset development and algorithm design, which restricts many applications such as depth estimation and object recognition. To overcome this shortcoming, a super-resolution task that reconstructs all-in-focus high-resolution (HR) LF images from low-resolution (LR) LF images is proposed by designing a large dataset and proposing a convolutional neural network (CNN) -based SR method. The dataset is constructed by using Blender software, consisting of 150 light field images used as training data, and 15 light field images used as validation and testing data. The proposed network is designed by proposing the dilated deformable convolutional network (DCN) -based feature extraction block and the LF subaperture image (SAI) Deblur-SR block. The experimental results demonstrate that the proposed method achieves more appealing results both quantitatively and qualitatively.

Automated Visual Inspection of Defects in Transparent Display Layers Using Light-Field 3-D Imaging

Since a display panel comprises multiple layered components, defects may occur within different layers through manufacturing processes. Traditional visual inspection systems with a 2D camera cannot identify the occurrence location among layers. Several 3D imaging technologies, such as CT, TSOM, and MRI, suffer from slow performance and a large form factor. In this work, we propose a novel visual inspection method to detect defects on a display panel using light-field 3D imaging. Without powering the target display panel, we first acquire the high-resolution depth information of defects located inside the transparent layers. We then convert the depth information to the object coordinate system to estimate the physical locations of defects. We automatically classify the types of defects and their layer locations along the depth axis in multiple transparent layers of the display panel. Lastly, our experimental results validate that our method can successfully detect and classify various display defects.

Optical flow-assisted multi-level fusion network for Light Field image angular reconstruction

Light Field (LF) imaging can record both the intensities and directions of light rays in a single exposure, which has received extensive attentions. However, the limited angular resolution becomes the primary bottleneck for the wide-spread applications of LF imaging. To this end, this paper proposes a novel optical flow-assisted multi-level fusion network for LF angular reconstruction. In our method, we propose to infer the multi-angular optical flows to explore long-range dependency of LF sub-aperture images (SAIs) for high-quality angular reconstruction. By aligning the SAIs in multi-angular directions, the geometric consistency of reconstructed dense LF can be preserved. Moreover, a multi-level fusion framework for LF angular reconstruction is introduced, which consists of two stages, namely texture-optical flow feature fusion and parallax structure-information fusion. The former firstly extracts the texture and optical flow features from the reconstructed coarse LF and then fuses these two features by using the proposed texture-optical flow fusion-block. The latter further blends the LF parallax structure information with the fused texture and optical flow features using the proposed parallax structure-information fusion network. Comprehensive experiments on both real-world and synthetic LF scenes demonstrate the superiority of the proposed method for reconstructing high-quality dense LF. Moreover, practical application on depth estimation also validates that our method can recover more texture details, particularly for some occlusion regions.

Progressive spatial-angular feature enhancement network for light field image super-resolution

The intensity and direction of the light field (LF) can be recorded simultaneously by using LF cameras. However, since LF cameras sacrifice spatial resolution for higher angular resolution, the images acquired by LF cameras tend to have low spatial resolution. Therefore, LF image super-resolution (SR) has become an integral part of LF studies. Many existing LF image SR methods fail to fully utilize angular and spatial information due to only using partial sub-aperture images (SAIs). In this paper, we propose a progressive spatial-angular feature enhancement network (PSAFENet) to deal with the problem of missing information in LF image SR. Specifically, we first extract the spatial features of SAIs, the spatial and angular features contained in the macro-pixel images (MacPIs) by three different feature extraction modules. Then, these features are fed into a spatial-angular feature enhancement (SAFE) module to perform enhancement of spatial-angular information on the SAIs. To improve the reconstruction accuracy, we also use the information multi-distillation block (IMDB) to remove the redundant information before upsampling. Our network can well merge the angular and spatial information into each SAI, which facilitates the reconstruction of the LF images. Experimental results on five public datasets show that the proposed PSAFENet method outperforms existing methods in both qualitative and quantitative comparisons.

Image quality enhancement of 4D light field microscopy via reference impge propagation-based one-shot learning

Four-dimensional (4D) light-field (LF) microscopes can acquire 3D information about target objects using a microlens array (MLA). However, the resolution and quality of sub-images in the LF images are reduced because of the spatial multiplexing of rays by the element lenses of the MLA. To overcome these limitations, this study proposes an LF one-shot learning technique that can convert LF sub-images into high-quality images similar to the 2D images of conventional optical microscopes obtained without any external training datasets for image enhancement. The proposed convolutional neural network model was trained using only one training dataset comprising a high-resolution reference image captured without an MLA as the ground truth. Further, its input was the central view of the LF image. After LF one-shot learning, the trained model should be able to convert well the other LF sub-images of various directional views that were not used in the main training process. Therefore, novel learning techniques were designed for LF one-shot learning. These novel techniques include an autoencoder-based model initialization method, a feature map-based learning algorithm to prevent the overfitting of the model, and cut loss to prevent saturation. The experimental results verified that the proposed technique effectively enhances the LF image quality and resolution using a reference image. Moreover, this method enhances the resolution by up to 13 times, decreases the noise amplification effect, and restores the lost details of microscopic objects. The proposed technique is stable and yields superior experimental results compared with those of the existing resolution-enhancing methods.

Comprehensive High-Quality Three-Dimensional Display System Based on a Simplified Light-Field Image Acquisition Method and a Full-Connected Deep Neural Network.

We propose a high-quality, three-dimensional display system based on a simplified light field image acquisition method, and a custom-trained full-connected deep neural network is proposed. The ultimate goal of the proposed system is to acquire and reconstruct the light field images with possibly the most elevated quality from the real-world objects in a general environment. A simplified light field image acquisition method acquires the three-dimensional information of natural objects in a simple way, with high-resolution/high-quality like multicamera-based methods. We trained a full-connected deep neural network model to output desired viewpoints of the object with the same quality. The custom-trained instant neural graphics primitives model with hash encoding output the overall desired viewpoints of the object within the acquired viewing angle in the same quality, based on the input perspectives, according to the pixel density of a display device and lens array specifications within the significantly short processing time. Finally, the elemental image array was rendered through the pixel re-arrangement from the entire viewpoints to visualize the entire field-of-view and re-constructed as a high-quality three-dimensional visualization on the integral imaging display. The system was implemented successfully, and the displayed visualizations and corresponding evaluated results confirmed that the proposed system offers a simple and effective way to acquire light field images from real objects with high-resolution and present high-quality three-dimensional visualization on the integral imaging display system.

Dynamic holographic stereogram using temporal and perspective coherence

A dynamic holographic stereogram (HS) rendering technique using temporal and perspective (TP) coherence is proposed and demonstrated. It solves the issue that previous holographic rendering methods are difficult to record HS of time-varying three-dimensional (3D) images efficiently. In our technique, a unified mathematical model is set in EPIpolar coordinates for the generation of light field images of dynamic HS. The time dimension is encoded in angular coordinates through angular multiplexing. Temporal coherence of dynamic 3D images is applied to simplify the rendering of action key frames. The action key frames are the same as conventional key frames, but have 3D information. Perspective coherence is used to render the light field images between two adjacent action key frames. Parallel computing technology combined with TP coherence is used to accelerate the rendering, which is on average 378.8% faster than traditional serial per-viewpoint rendering (PVR) algorithms. HSs with vivid and striking 3D animations are demonstrated using a custom holographic printing system, which validates the effectiveness of the proposed method. The results pave the way towards developing high-performance dynamic HSs that can be used in 3D display, visual security, and virtual/augmented reality domain.

Light field super-resolution using complementary-view feature attention

Light field (LF) cameras record multiple perspectives by a sparse sampling of real scenes, and these perspectives provide complementary information. This information is beneficial to LF super-resolution (LFSR). Compared with traditional single-image super-resolution, LF can exploit parallax structure and perspective correlation among different LF views. Furthermore, the performance of existing methods are limited as they fail to deeply explore the complementary information across LF views. In this paper, we propose a novel network, called the light field complementary-view feature attention network (LF-CFANet), to improve LFSR by dynamically learning the complementary information in LF views. Specifically, we design a residual complementary-view spatial and channel attention module (RCSCAM) to effectively interact with complementary information between complementary views. Moreover, RCSCAM captures the relationships between different channels, and it is able to generate informative features for reconstructing LF images while ignoring redundant information. Then, a maximum-difference information supplementary branch (MDISB) is used to supplement information from the maximum-difference angular positions based on the geometric structure of LF images. This branch also can guide the process of reconstruction. Experimental results on both synthetic and real-world datasets demonstrate the superiority of our method. The proposed LF-CFANet has a more advanced reconstruction performance that displays faithful details with higher SR accuracy than state-of-the-art methods.

Take Your Model Further: A General Post-refinement Network for Light Field Disparity Estimation via BadPix Correction

Most existing light field (LF) disparity estimation algorithms focus on handling occlusion, texture-less or other areas that harm LF structure to improve accuracy, while ignoring other potential modeling ideas. In this paper, we propose a novel idea called Bad Pixel (BadPix) correction for method modeling, then implement a general post-refinement network for LF disparity estimation: Bad-pixel Correction Network (BpCNet). Given an initial disparity map generated by a specific algorithm, we assume that all BadPixs on it are in a small range. Then BpCNet is modeled as a fine-grained search strategy, and a more accurate result can be obtained by evaluating the consistency of LF images in this limited range. Due to the assumption and the consistency between input and output, BpCNet can perform as a general post-refinement network, and can work on almost all existing algorithms iteratively. We demonstrate the feasibility of our theory through extensive experiments, and achieve remarkable performance on the HCI 4D Light Field Benchmark.