Light Field Synthesis Research Articles

Robust Local Light Field Synthesis via Occlusion-aware Sampling and Deep Visual Feature Fusion

Novel view synthesis has attracted tremendous research attention recently for its applications in virtual reality and immersive telepresence. Rendering a locally immersive light field (LF) based on arbitrary large baseline RGB references is a challenging problem that lacks efficient solutions with existing novel view synthesis techniques. In this work, we aim at truthfully rendering local immersive novel views/LF images based on large baseline LF captures and a single RGB image in the target view. To fully explore the precious information from source LF captures, we propose a novel occlusion-aware source sampler (OSS) module which efficiently transfers the pixels of source views to the target view’s frustum in an occlusion-aware manner. An attention-based deep visual fusion module is proposed to fuse the revealed occluded background content with a preliminary LF into a final refined LF. The proposed source sampling and fusion mechanism not only helps to provide information for occluded regions from varying observation angles, but also proves to be able to effectively enhance the visual rendering quality. Experimental results show that our proposed method is able to render high-quality LF images/novel views with sparse RGB references and outperforms state-of-the-art LF rendering and novel view synthesis methods.

Light Field View Synthesis Using the Focal Stack and All-in-Focus Image.

Light field reconstruction and synthesis algorithms are essential for improving the lower spatial resolution for hand-held plenoptic cameras. Previous light field synthesis algorithms produce blurred regions around depth discontinuities, especially for stereo-based algorithms, where no information is available to fill the occluded areas in the light field image. In this paper, we propose a light field synthesis algorithm that uses the focal stack images and the all-in-focus image to synthesize a 9 × 9 sub-aperture view light field image. Our approach uses depth from defocus to estimate a depth map. Then, we use the depth map and the all-in-focus image to synthesize the sub-aperture views, and their corresponding depth maps by mimicking the apparent shifting of the central image according to the depth values. We handle the occluded regions in the synthesized sub-aperture views by filling them with the information recovered from the focal stack images. We also show that, if the depth levels in the image are known, we can synthesize a high-accuracy light field image with just five focal stack images. The accuracy of our approach is compared with three state-of-the-art algorithms: one non-learning and two CNN-based approaches, and the results show that our algorithm outperforms all three in terms of PSNR and SSIM metrics.

Petahertz-scale spectral broadening and few-cycle compression of Yb:KGW laser pulses in a pressurized, gas-filled hollow-core fiber.

We demonstrate efficient generation of coherent super-octave pulses via a single-stage spectral broadening of a Yb:KGW laser in a single, pressurized, Ne-filled, hollow-core fiber capillary. Emerging pulses spectrally spanning over more than 1 PHz (250-1600 nm) at a dynamic range of ∼60 dB, and an excellent beam quality open the door to combining Yb:KGW lasers with modern light-field synthesis techniques. Compression of a fraction of the generated supercontinuum to intense (8 fs, ∼2.4 cycle, ∼650 µJ) pulses allows convenient use of these novel laser sources in strong-field physics and attosecond science.

20.3: Weighted Simultaneous Algebra Reconstruction Technique (wSART) for Additive Light Field Synthesis

We apply an iterative weighting scheme for additive light field synthesis. Unlike previous work optimizing additive light field evenly over viewpoints, we constrain the optimization to deliver a reconstructed light field of high image quality for viewpoints of large weight.

Attosecond light field synthesis

The advancement of the ultrafast pulse shaping and waveform synthesis allowed to coherently control the atomic and electronic motions in matter. The temporal resolution of the waveform synthesis is inversely proportional to the broadening of its spectrum. Here, we demonstrate the light field synthesis of high-power waveforms spanning two optical octaves, from near-infrared to deep-ultraviolet with attosecond resolution. Moreover, we utilized the all-optical field sampling metrology for on-demand tailoring of light field waveforms to control the electron motion in matter. The demonstrated synthesis of the light field and the electron motion control pave the way for switching the photo-induced current signal in dielectric nanocircuit and establishing ultrafast photonics operating beyond the petahertz speed.

Exploring Spatial Correlation for Light Field Saliency Detection: Expansion From a Single View.

Previous 2D saliency detection methods extract salient cues from a single view and directly predict the expected results. Both traditional and deep-learning-based 2D methods do not consider geometric information of 3D scenes. Therefore the relationship between scene understanding and salient objects cannot be effectively established. This limits the performance of 2D saliency detection in challenging scenes. In this paper, we show for the first time that saliency detection problem can be reformulated as two sub-problems: light field synthesis from a single view and light-field-driven saliency detection. This paper first introduces a high-quality light field synthesis network to produce reliable 4D light field information. Then a novel light-field-driven saliency detection network is proposed, in which a Direction-specific Screening Unit (DSU) is tailored to exploit the spatial correlation among multiple viewpoints. The whole pipeline can be trained in an end-to-end fashion. Experimental results demonstrate that the proposed method outperforms the state-of-the-art 2D, 3D and 4D saliency detection methods. Our code is publicly available at https://github.com/OIPLab-DUT/ESCNet.

Scale-Consistent Fusion: From Heterogeneous Local Sampling to Global Immersive Rendering.

Image-based geometric modeling and novel view synthesis based on sparse large-baseline samplings are challenging but important tasks for emerging multimedia applications such as virtual reality and immersive telepresence. Existing methods fail to produce satisfactory results due to the limitation on inferring reliable depth information over such challenging reference conditions. With the popularization of commercial light field (LF) cameras, capturing LF images (LFIs) is as convenient as taking regular photos, and geometry information can be reliably inferred. This inspires us to use a sparse set of LF captures to render high-quality novel views globally. However, the fusion of LF captures from multiple angles is challenging due to the scale inconsistency caused by various capture settings. To overcome this challenge, we propose a novel scale-consistent volume rescaling algorithm that robustly aligns the disparity probability volumes (DPV) among different captures for scale-consistent global geometry fusion. Based on the fused DPV projected to the target camera frustum, novel learning-based modules (i.e., the attention-guided multi-scale residual fusion module, and the disparity field-guided deep re-regularization module), which comprehensively regularize noisy observations from heterogeneous captures for high-quality rendering of novel LFIs, have been proposed. Both quantitative and qualitative experiments over the Stanford Lytro Multi-view LF dataset show that the proposed method outperforms state-of-the-art methods significantly under different experiment settings for disparity inference and LF synthesis.

Advances in Confocal Microscopy and Selected Applications.

Over the last 30 years, confocal microscopy has emerged as a primary tool for biological investigation across many disciplines. The simplicity of use and widespread accessibility of confocal microscopy ensure that it will have a prominent place in biological imaging for many years to come, even with the recent advances in light sheet and field synthesis microscopy. Since these more advanced technologies still require significant expertise to effectively implement and carry through to analysis, confocal microscopy-based approaches still remain the easiest way for biologists with minimal imaging experience to address fundamental questions about how their systems are arranged through space and time. In this review, we discuss a number of advanced applications of confocal microscopy for probing the spatiotemporal dynamics of biological systems.

Light Field Synthesis by Training Deep Network in the Refocused Image Domain.

Light field imaging, which captures spatial-angular information of light incident on image sensors, enables many interesting applications such as image refocusing and augmented reality. However, due to the limited sensor resolution, a trade-off exists between the spatial and angular resolutions. To increase the angular resolution, view synthesis techniques have been adopted to generate new views from existing views. However, traditional learning-based view synthesis mainly considers the image quality of each view of the light field and neglects the quality of the refocused images. In this paper, we propose a new loss function called refocused image error (RIE) to address the issue. The main idea is that the image quality of the synthesized light field should be optimized in the refocused image domain because it is where the light field is viewed. We analyze the behavior of RIE in the spectral domain and test the performance of our approach against previous approaches on both real (INRIA) and software-rendered (HCI) light field datasets using objective assessment metrics such as MSE, MAE, PSNR, SSIM, and GMSD. Experimental results show that the light field generated by our method results in better refocused images than previous methods.

利用光场合成与弥散圆渲染的单幅图像重聚焦

利用光场合成与弥散圆渲染的单幅图像重聚焦

Computational Super-Resolution Full-Parallax Three-Dimensional Light Field Display Based on Dual-Layer LCD Modulation

Due to physical limitation in resolution of display device, it is usually very difficult to achieve a high spatial resolution and a high angular resolution simultaneously for a three-dimensional (3D) light field display. Here, a computational super-resolution full-parallax 3D light field display is demonstrated, which can achieve both high spatial resolution and angular resolution. The proposed display consists of a specially designed backlight unit with controlled scattering angle, two cascadedly arranged LCDs with a resolution of 3840 × 2160, an aberration-suppressed compound lens array, and a holographic functional screen. The optical image formation processing of the display is analyzed and modeled as a whole process, and the superresolution light field synthesis method is presented. By co-designing the optical elements with computation processing, our proposed display architecture can improve the spatial resolution and angular resolution with a factor of 2× in both horizontal and vertical directions, comparing with these of the conventional light field display with single LCD and standard lens array. Finally, a super-resolution full-parallax 3D light field display with a designed spatial resolution of about 640 × 360 in a display area of 21.5 inches and an angular resolution of 166 × 166 in a viewing angle of 45 × 45 degrees is constructed, and an excellent 3D visual experience with improved image quality can be perceived.

Time-multiplexed light field synthesis via factored Wigner distribution function.

An optimization algorithm for preparing display-ready holographic elements (hogels) to synthesize a light field is outlined, and proof of concept is experimentally demonstrated. This method allows for higher-rank factorization, which can be used for time-multiplexing multiple frames for improved image quality, using phase-only and fully complex modulation with a single spatial light modulator.

Transforming the Commonplace through Machine Perception: Light Field Synthesis and Audio Feature Extraction in the Rover Project

Rover is a mechatronic imaging device inserted into quotidian space, transforming the sights and sounds of the everyday through its peculiar modes of machine perception. Using computational light field photography and machine listening, it creates a kind of cinema following the logic of dreams: suspended but mobile, familiar yet infinitely variable in detail. Rover draws on diverse traditions of robotic exploration, landscape and still-life depiction, and audio field recording to create a hybrid form between photography and cinema. This paper describes the mechatronic, machine perception, and audio-visual synthesis techniques developed for the piece.

Holographic display system for dynamic synthesis of 3D light fields with increased space bandwidth product.

We present a new method for the generation of a dynamic wave field with high space bandwidth product (SBP). The dynamic wave field is generated from several wave fields diffracted by a display which comprises multiple spatial light modulators (SLMs) each having a comparably low SBP. In contrast to similar approaches in stereoscopy, we describe how the independently generated wave fields can be coherently superposed. A major benefit of the scheme is that the display system may be extended to provide an even larger display. A compact experimental configuration which is composed of four phase-only SLMs to realize the coherent combination of independent wave fields is presented. Effects of important technical parameters of the display system on the wave field generated across the observation plane are investigated. These effects include, e.g., the tilt of the individual SLM and the gap between the active areas of multiple SLMs. As an example of application, holographic reconstruction of a 3D object with parallax effects is demonstrated.

A compressive light field projection system

For about a century, researchers and experimentalists have strived to bring glasses-free 3D experiences to the big screen. Much progress has been made and light field projection systems are now commercially available. Unfortunately, available display systems usually employ dozens of devices making such setups costly, energy inefficient, and bulky. We present a compressive approach to light field synthesis with projection devices. For this purpose, we propose a novel, passive screen design that is inspired by angle-expanding Keplerian telescopes. Combined with high-speed light field projection and nonnegative light field factorization, we demonstrate that compressive light field projection is possible with a single device. We build a prototype light field projector and angle-expanding screen from scratch, evaluate the system in simulation, present a variety of results, and demonstrate that the projector can alternatively achieve super-resolved and high dynamic range 2D image display when used with a conventional screen.

Efficient Reconstruction of All-in-Focus Images Through Shifted Pinholes From Multi-Focus Images for Dense Light Field Synthesis and Rendering

Scene refocusing beyond extended depth of field for users to observe objects effectively is aimed by researchers in computational photography, microscopic imaging, and so on. Ordinary all-in-focus image reconstruction from a sequence of multi-focus images achieves extended depth of field, where reconstructed images would be captured through a pinhole in the center on the lens. In this paper, we propose a novel method for reconstructing all-in-focus images through shifted pinholes on the lens based on 3D frequency analysis of multi-focus images. Such shifted pinhole images are obtained by a linear combination of multi-focus images with scene-independent 2D filters in the frequency domain. The proposed method enables us to efficiently synthesize dense 4D light field on the lens plane for image-based rendering, especially, robust scene refocusing with arbitrary bokeh. Our novel method using simple linear filters achieves not only reconstruction of all-in-focus images even for shifted pinholes more robustly than the conventional methods depending on scene/focus estimation, but also scene refocusing without suffering from limitation of resolution in comparison with recent approaches using special devices such as lens arrays in computational photography.

Real-time Image Generation for Compressive Light Field Displays

With the invention of integral imaging and parallax barriers in the beginning of the 20th century, glasses-free 3D displays have become feasible. Only today—more than a century later—glasses-free 3D displays are finally emerging in the consumer market. The technologies being employed in current-generation devices, however, are fundamentally the same as what was invented 100 years ago. With rapid advances in optical fabrication, digital processing power, and computational perception, a new generation of display technology is emerging: compressive displays exploring the co-design of optical elements and computational processing while taking particular characteristics of the human visual system into account. In this paper, we discuss real-time implementation strategies for emerging compressive light field displays. We consider displays composed of multiple stacked layers of light-attenuating or polarization-rotating layers, such as LCDs. The involved image generation requires iterative tomographic image synthesis. We demonstrate that, for the case of light field display, computed tomographic light field synthesis maps well to operations included in the standard graphics pipeline, facilitating efficient GPU-based implementations with real-time framerates.

Synthesis of three-dimensional light fields with binary spatial light modulators

Computation of a binary spatial light modulator (SLM) pattern that generates a desired light field is a challenging quantization problem for which several algorithms have been proposed, mainly for far-field or Fourier plane reconstructions. We study this problem assuming that the desired light field is synthesized within a volumetric region in the non-far-field range after free space propagation from the SLM plane. We use Fresnel and Rayleigh-Sommerfeld scalar diffraction theories for propagation of light. We show that, when the desired field is confined to a sufficiently narrow region of space, the ideal gray-level complex-valued SLM pattern generating it becomes sufficiently low pass (oversampled) so it can be successfully halftoned into a binary SLM pattern by solving two decoupled real-valued constrained halftoning problems. Our simulation results indicate that, when the synthesis region is considered, the binary SLM is indistinguishable from a lower resolution full complex gray-level SLM. In our approach, free space propagation related computations are done only once at the beginning, and the rest of the computation time is spent on carrying out standard image halftoning.

Synthesis of three-dimensional light fields and applications [Prolog

Synthesis of three-dimensional light fields and applications [Prolog

Synthesis of three-dimensional light fields and applications

The possibility of synthesizing light fields satisfying given requirements within a three-dimensional (3-D) space domain was proposed and demonstrated during recent years. In this paper we present fundamental physical properties characterizing 3-D fields and propose analytical and numerical procedures to synthesize them. These methods solve the proper wave equation under 3-D constraints. Since the manipulation of light is a basic task, this is an effective approach to deal with classical as well as novel problems in optics and photonics technologies. We discuss exciting new potential areas of application and extensions of this concept.