Light Field Technology Research Articles

Pixel-wise matching cost function for robust light field depth estimation

Depth estimation has significant potential in industrial applications, such as robot navigation. Light Field (LF) technology captures both spatial and angular information of a scene, enabling precise depth acquisition. Stereo matching technology has been commonly used for this task in LF imaging. However, previous methods typically compute the matching cost by simply shifting images with predefined offsets, causing poor performance in regions with sudden depth distribution changes (e.g., edges). To address this issue, we propose a Pixel-wise Matching Cost Function (PMCF) to estimate depth in units of pixel, then design a pixel-based network, termed PixelNet, for depth prediction, which achieves top rankings on the HCI 4D light field benchmark. Specifically, our cost function consists of two modules: the range search strategy and the modulation mechanism. The range search strategy enables the network to iteratively optimize depth on a per-pixel basis, while the modulation mechanism effectively handles scene noise and occlusions. By integrating these two modules, our function enables the generation of depth maps with sharper edges and smoother surfaces, compared with previous methods. Finally, we design a pixel-based network and apply it to both synthetic and real-world scenes, demonstrating that our method outperforms state-of-the-art methods in both scenes. Furthermore, PixelNet can also function as a post-processing network that can be seamlessly integrated with existing methods for depth refinement.

Chromatic correction for super multi-view 3D display based on diffraction theory.

The traditional analysis method for super multi-view 3D display based on geometric optics, which approximates the lenticular lenses as a series of pinhole structures, ignored the chromatic aberration. In this paper, the optimization method based on diffraction theory is proposed for super multi-view 3D display, where the wavefronts are evaluated accurately by the forward propagation method, and the chromatic aberration of the synthetic viewpoint image is reduced dramatically by the backward reconstruction optimization method (BROM). The optical experiment is performed to verify the feasibility of the method, which is consistent with numerical simulation results. It is proved that the proposed method simulates the physical propagation process of super multi-view 3D display and improves the reconstructed image quality. In the future, it can be used to achieve the super multi-view 3D light field technology with low crosstalk.

Multi‐cues underwater image restoration algorithm combined with light field technology

AbstractUnderwater images suffer from color distortion, low clarity and halos problems due to light absorption, particle scattering and non‐uniform illumination. To address these degradation issues, a multi‐cues underwater image restoration algorithm combined with light field technology is proposed. First, based on Epipolar Plane Image, the light field cue transmittance containing depth information is calculated. Then, according to the turbidity of the underwater image, the light field cue transmittance and the polarization cue transmittance are fused to obtain the multi‐cues transmittance, which can effectively reduce the effect of particle scattering and color bias. Finally, the background light is estimated through the all‐focus operation, which can effectively overcome the distortion of an underwater single image and simultaneously reduce the halo phenomenon. Experimental results show that the method achieves the best results evaluated by UCIQE, UIQM, PSNR, and SSIM, and the restored color under the method is closer to the actual image than other underwater restoration methods.

19‐2: A Novel 3D Floating Image for Human‐Machine Interfaces and Its Application

Recently, coronavirus disease (COVID‐19) has caused massive loss of human life and property worldwide. Many people are devoting a lot of effort in doing research to effectively block the route of infection. For this reason, we developed a novel noncontact human‐machine interfaces (HMIs) that integrate light field technology and non‐contacting touch sensing to apply the commonly public facilities. Moreover, the methodology of lightfield technology which provided the 3D floating image, multiple perspectives, and naked‐eyes type is also introduced.

How I Met Your V2X Sensor Data: Analysis of Projection-Based Light Field Visualization for Vehicle-to-Everything Communication Protocols and Use Cases.

The practical usage of V2X communication protocols started emerging in recent years. Data built on sensor information are displayed via onboard units and smart devices. However, perceptually obtaining such data may be counterproductive in terms of visual attention, particularly in the case of safety-related applications. Using the windshield as a display may solve this issue, but switching between 2D information and the 3D reality of traffic may introduce issues of its own. To overcome such difficulties, automotive light field visualization is introduced. In this paper, we investigate the visualization of V2X communication protocols and use cases via projection-based light field technology. Our work is motivated by the abundance of V2X sensor data, the low latency of V2X data transfer, the availability of automotive light field prototypes, the prevalent dominance of non-autonomous and non-remote driving, and the lack of V2X-based light field solutions. As our primary contributions, we provide a comprehensive technological review of light field and V2X communication, a set of recommendations for design and implementation, an extensive discussion and implication analysis, the exploration of utilization based on standardized protocols, and use-case-specific considerations.

Learning a Compact Spatial-Angular Representation for Light Field

The recent emergence of light field technology has led to new opportunities for immersive visual communication that has a need for high spatial and angular resolution, both of which contribute to a large image storage footprint and high-latency transmission. Task-driven downsampling methods have been proposed as a solution, and have shown improvements in single-image restoration. However, they are inevitable to disregard light field's intrinsic properties in the corresponding tasks. In this paper, we propose a light-field-specific task-driven downsampling framework, called the LFCrNet. The LFCrNet operates on a learning-based decreasing and increasing resolution in an end-to-end manner in order to utilize a cross-view asymmetric sampling technique. In detail, it separates raw data into disparity and non-disparity patterns by measuring pixel-wise residuals between the sub-aperture central view and auxiliary views. Then, a chain of three-dimensional (3-D) deformable residual blocks (DRBs) is used to extract disparity features and manage these features regard of their intrinsic property individually. Afterwards, they are compacted into spatio-angular domains through a 3-D deformable downsampler (3-DDS). The non-disparity information is integrated into a separate pipeline that leverages spatial similarity across multiple light field views. This technique is capable of preserving specific occlusion components, and subsequently, restoring them using a learning-based upscaling method to generate a high-quality reconstruction. In general, our method has shown superior performance on multiple open-source datasets by a significant margin.

Virtual stereo content rendering technology review for light-field display

Light field display (LFD) is considered as a promising technology to reconstruct the light rays’ distribution of the real 3D scene, which approximates the original light field of target displayed objects with all depth cues in human vision including binocular disparity, motion parallax, color hint and correct occlusion relationship. Currently, computer-generated content is widely used for the LFD system, therefore rich 3D content can be provided. This paper firstly introduces applications of light field technologies in display system. Additionally, virtual stereo content rendering techniques and their application scenes are thoroughly combed and pointed out its pros and cons. Moreover, according to the different characteristics of light field system, the coding and correction algorithms in virtual stereo content rendering techniques are reviewed. Through the above discussion, there are still many problems in the existing rendering techniques for LFD. New rendering algorithms should be introduced to solve the real-time light-field rendering problem for large-scale virtual scenes.

54.3: Invited Paper: Surface Control of LC Alignment for Creation of Liquid Crystal Lenses Arrays

New photosensitive alignment materials, capable to obtain pretilt angle from 90 to 0 degree in single technological process of UV exposure, are developed. The novel technological method for fabrication LC lens and LC lenses arrays has been developed based on new pretilt angle gradient alignment materials. Laboratory samples of voltage tunable LC lens and LC lenses arrays are obtained and characterized. The developed low cost fabrication technology of tunable micro‐lenses arrays based on photosensitive alignment materials has low complexity and is perspective for light field technology applications.

End-to-End Residual Network for Light Field Reconstruction on Raw Images and View Image Stacks

Light field (LF) technology has become a focus of great interest (due to its use in many applications), especially since the introduction of the consumer LF camera, which facilitated the acquisition of dense LF images. Obtaining densely sampled LF images is costly due to the trade-off between spatial and angular resolutions. Accordingly, in this research, we suggest a learning-based solution to this challenging problem, reconstructing dense, high-quality LF images. Instead of training our model with several images of the same scene, we used raw LF images (lenslet images). The raw LF format enables the encoding of several images of the same scene into one image. Consequently, it helps the network to understand and simulate the relationship between different images, resulting in higher quality images. We divided our model into two successive modules: LFR and LF augmentation (LFA). Each module is represented using a convolutional neural network-based residual network (CNN). We trained our network to lessen the absolute error between the novel and reference views. Experimental findings on real-world datasets show that our suggested method has excellent performance and superiority over state-of-the-art approaches.

Preview the Products on Display at Display Week 2022

Preview the Products on Display at Display Week 2022

GaN-based mini-LED matrix applied to multi-functional forward lighting

In this paper, we propose and demonstrate to use of a single reflector with 68 segments to project vehicle low beam and high beam with the use of a GaN-based mini-LED matrix, which is a 5 × 6 LED die array. The design of the reflector is based on light field technology in considering etendue from the light source across the segments. The group of the segments with smaller etendue from the LED dies in the bottom 2 rows are used to project low beams. When the other LED dies are turned on, the reflector will project light upward and form the high beam. The selection of the turn-on LED dies in the mini-LED matrix can adjust the width of the illumination pattern so that an adaptive low/high beam can be performed. Besides, to extend the functionality of the headlamp module, we propose to dispense IR phosphor on LED dies in the high-beam zone of the GaN-based mini-LED matrix. Thus the vehicle can emit IR high beam, which can be imaged through a camera and can be incorporated with machine vision for an autonomous vehicle without using a complicated adaptive headlight to avoid glare. The proposed multi-function in spatial and spectral domains will be helpful to various applications with use of a mini-LED matrix.

Hierarchical lossless coding of light fields with improved random access

The adoption of light field (LF) technology is often hindered by the large amount of data that is necessary to represent such information. To alleviate such burden, specific coding techniques have been under research in recent years. However, most of them require decoding the whole LF data, even in applications where only a region of the LF visual content needs to be accessed. Therefore, encoding schemes that incorporate scalability and random access capabilities would cater to a wider range of LF use cases, particularly those requiring lossless compression. To enable such type of functionalities, an encoder must be designed so that the decoding process can extract only specific views/regions of the complete LF, according to user options or application requirements. This paper proposes hierarchical Minimum Rate Predictors (H-MRP), a lossless light field codec that provides the aforementioned functionalities. The main contributions herein proposed are: (i) the lossless encoding the LF in multiple hierarchical layers; (ii) the flexible selection of reference images for prediction; and (iii) the use disparity compensation between the reference images. The experimental results show that the proposed encoder provides a good compromise in terms of compression efficiency versus random access penalty, surpassing its best competitor by 25% in terms of random access and by 1 bit-per-pixel in terms of compression efficiency. When optimised to provide the best compression efficiency, the method surpasses current state-of-the-art lossless encoding by up to 3 bits-per-pixel.

Light Field Reconstruction Using Residual Networks on Raw Images.

Although Light-Field (LF) technology attracts attention due to its large number of applications, especially with the introduction of consumer LF cameras and its frequent use, reconstructing densely sampled LF images represents a great challenge to the use and development of LF technology. Our paper proposes a learning-based method to reconstruct densely sampled LF images from a sparse set of input images. We trained our model with raw LF images rather than using multiple images of the same scene. Raw LF can represent the two-dimensional array of images captured in a single image. Therefore, it enables the network to understand and model the relationship between different images of the same scene well and thus restore more texture details and provide better quality. Using raw images has transformed the task from image reconstruction into image-to-image translation. The feature of small-baseline LF was used to define the images to be reconstructed using the nearest input view to initialize input images. Our network was trained end-to-end to minimize the sum of absolute errors between the reconstructed and ground-truth images. Experimental results on three challenging real-world datasets demonstrate the high performance of our proposed method and its outperformance over the state-of-the-art methods.

Enhancement of light field disparity maps by reducing the silhouette effect and plane noise

During the last decade, there has been an increasing number of applications dealing with multidimensional visual information, either for 3D object representation or feature extraction purposes. In this context, recent advances in light field technology, have been driving research efforts in disparity estimation methods. Among the existing ones, those based on the structure tensor have emerged as very promising to estimate disparity maps from Epipolar Plane Images. However, this approach is known to have two intrinsic limitations: (i) silhouette enlargement and (ii) irregularity of surface normal maps as computed from the estimated disparity. To address these problems, this work proposes a new method for improving disparity maps obtained from the structure-tensor approach by enhancing the silhouette and reducing the noise of planar surfaces in light fields. An edge-based approach is initially used for silhouette improvement through refinement of the estimated disparity values around object edges. Then, a plane detection algorithm, based on a seed growth strategy, is used to estimate planar regions, which in turn are used to guide correction of erroneous disparity values detected in object boundaries. The proposed algorithm shows an average improvement of 98.3% in terms of median angle error for plane surfaces, when compared to regular structure-tensor-based methods, outperforming state-of-the-art methods. The proposed framework also presents very competitive results, in terms of mean square error between disparity maps and their ground truth, when compared with their counterparts.

3D retinal imaging and measurement using light field technology.

.Significance: Light-field fundus photography has the potential to be a new milestone in ophthalmology. Up-to-date publications show only unsatisfactory image quality, preventing the use of depth measurements. We show that good image quality and, consequently, reliable depth measurements are possible, and we investigate the current challenges of this novel technology.Aim: We investigated whether light field (LF) imaging of the retina provides depth information, on which structures the depth is estimated, which illumination wavelength should be used, whether deeper layers are measurable, and what kinds of artifacts occur.Approach: The technical setup, a mydriatic fundus camera with an LF imager, and depth estimation were validated by an eye model and in vivo measurements of three healthy subjects and three subjects with suspected glaucoma. Comparisons between subjects and the corresponding optical coherence tomography (OCT) measurements were used for verification of the depth estimation.Results: This LF setup allowed for three-dimensional one-shot imaging and depth estimation of the optic disc with green light. In addition, a linear relationship was found between the depth estimates of the OCT and those of the setup developed here. This result is supported by the eye model study. Deeper layers were not measurable.Conclusions: If image artifacts can be handled, LF technology has the potential to help diagnose and monitor glaucoma risk at an early stage through a rapid, cost-effective one-shot technology.

Research of Light Field Technology

Light field technology is a relatively new theory and technical direction. At present, it has played a significant role in scientific research, production and life, national defense and military, but there are still many problems to be solved in terms of theoretical description, technical implementation, software and hardware processing capabilities, commercialization costs, and ease of use. Therefore, research the light field technology, explore the principle and application of light field imaging, and put forward some enlightenments for its application in industrial machine vision.

A new dimension for machine vision

AbstractAutomation is central to producing high quality products in high‐wage countries with an omnipresent shortage of skilled workers. Automation is becoming increasingly more complex due to the higher flexibility in produced goods. At the same time, this places high demands on integrated image processing and, for this reason, machine vision solutions have so far often fallen short of expectations. The use of 3D light field technology can help to solve demanding tasks quickly, easily and reliably.

Development of a scalable tabletop display using projection-based light field technology

We present a light-field display optical system design, state-of-the-art hardware technology, and computational software that are readily scalable (due to their modular structure) and provide naturally immersive and volumetric display systems. We explain the integration of 72 microprojectors into the tabletop display system. In addition, we used 6 workstations for all images and 18 high-performance graphics processing units. Sophisticated image generation through author tooling and the pixel re-arrangement algorithm in the Unity engine enabled us to produce complete three-dimensional images from the radiant rays of two-dimensional pixels using diffuser screens and microlens arrays.

Underwater Imaging System Based on Light Field Technology

The image quality might be degraded severely due to water scatterings, especially near-shore water scatterings. In addition, the degraded image reduces the accuracy of underwater 3D reconstruction. We propose an underwater imaging system based on light field camera array to obtain enhanced high-quality images and accurate three-dimensional reconstruction from harsh underwater environments. Both 2D-imaging and 3D-reconstruction using light-field imaging were studied by experiments. By refocusing and analyzing the light field images of the target through turbid water, we obtained accurate three-dimensional estimations in the underwater scene. These experiments confirm the reliability of our method, showing that the SSIM (structural similarity) value of the system in turbid water is 0.378, compared to 0.197, the SSIM value of conventional single-camera imaging in the experiments. Therefore, we verify that the light field imaging system is less affected by forward scattering of water and has a larger detective range than conventional single-camera imaging with any specific resolution.

Trans-illumination intestine projection imaging of intestinal motility in mice

Functional intestinal imaging holds importance for the diagnosis and evaluation of treatment of gastrointestinal diseases. Currently, preclinical imaging of intestinal motility in animal models is performed either invasively with excised intestines or noninvasively under anesthesia, and cannot reveal intestinal dynamics in the awake condition. Capitalizing on near-infrared optics and a high-absorbing contrast agent, we report the Trans-illumination Intestine Projection (TIP) imaging system for free-moving mice. After a complete system evaluation, we performed in vivo studies, and obtained peristalsis and segmentation motor patterns of free-moving mice. We show the in vivo typical segmentation motor pattern, that was previously shown in ex vivo studies to be controlled by intestinal pacemaker cells. We also show the effects of anesthesia on motor patterns, highlighting the possibility to study the role of the extrinsic nervous system in controlling motor patterns, which requires unanesthetized live animals. Combining with light-field technologies, we further demonstrated 3D imaging of intestine in vivo (3D-TIP). Importantly, the added depth information allows us to extract intestines located away from the abdominal wall, and to quantify intestinal motor patterns along different directions. The TIP system should open up avenues for functional imaging of the GI tract in conscious animals in natural physiological states.