Non-line-of-sight Imaging Research Articles

Real-time non-line-of-sight computational imaging using spectrum filtering and motion compensation.

Non-line-of-sight (NLOS) imaging aims at recovering the shape and albedo of hidden objects. Despite recent advances, real-time video of complex and dynamic scenes remains a major challenge owing to the weak signal of multiply scattered light. Here we propose and demonstrate a framework of spectrum filtering and motion compensation to realize high-quality NLOS video for room-sized scenes. Spectrum filtering leverages a wave-based model for denoising and deblurring in the frequency domain, enabling computational image reconstruction with a small number of sampling points. Motion compensation tailored with an interleaved scanning scheme can compute high-resolution live video during the acquisition of low-quality image sequences. Together, we demonstrate live NLOS videos at 4 fps for a variety of dynamic real-life scenes. The results mark a substantial stride toward real-time, large-scale and low-power NLOS imaging and sensing applications.

Influence of Target Surface BRDF on Non-Line-of-Sight Imaging

The surface material of an object is a key factor that affects non-line-of-sight (NLOS) imaging. In this paper, we introduce the bidirectional reflectance distribution function (BRDF) into NLOS imaging to study how the target surface material influences the quality of NLOS images. First, the BRDF of two surface materials (aluminized insulation material and white paint board) was modeled using deep neural networks and compared with a five-parameter empirical model to validate the method’s accuracy. The method was then applied to fit BRDF data for different common materials. Finally, NLOS target simulations with varying surface materials were reconstructed using the confocal diffusion tomography algorithm. The reconstructed NLOS images were classified via a convolutional neural network to assess how different surface materials impacted imaging quality. The results show that image clarity improves when decreasing the specular reflection and increasing the diffuse reflection, with the best results obtained for surfaces exhibiting a high diffuse reflection and no specular reflection.

Path Tracing-Inspired Modeling of Non-Line-of-Sight SPAD Data.

Non-Line of Sight (NLOS) imaging has gained attention for its ability to detect and reconstruct objects beyond the direct line of sight, using scattered light, with applications in surveillance and autonomous navigation. This paper presents a versatile framework for modeling the temporal distribution of photon detections in direct Time of Flight (dToF) Lidar NLOS systems. Our approach accurately accounts for key factors such as material reflectivity, object distance, and occlusion by utilizing a proof-of-principle simulation realized with the Unreal Engine. By generating likelihood distributions for photon detections over time, we propose a mechanism for the simulation of NLOS imaging data, facilitating the optimization of NLOS systems and the development of novel reconstruction algorithms. The framework allows for the analysis of individual components of photon return distributions, yielding results consistent with prior experimental data and providing insights into the effects of extended surfaces and multi-path scattering. We introduce an optimized secondary scattering approach that captures critical multi-path information with reduced computational cost. This work provides a robust tool for the design and improvement of dToF SPAD Lidar-based NLOS imaging systems.

Deep-Learning-Based Real-Time Passive Non-Line-of-Sight Imaging for Room-Scale Scenes.

Non-line-of-sight imaging is a technique for reconstructing scenes behind obstacles. We report a real-time passive non-line-of-sight (NLOS) imaging method for room-scale hidden scenes, which can be applied to smart home security monitoring sensing systems and indoor fast fuzzy navigation and positioning under the premise of protecting privacy. An unseen scene encoding enhancement network (USEEN) for hidden scene reconstruction is proposed, which is a convolutional neural network designed for NLOS imaging. The network is robust to ambient light interference conditions on diffuse reflective surfaces and maintains a fast reconstruction speed of 12.2 milliseconds per estimation. The consistency of the mean square error (MSE) is verified, and the peak signal-to-noise ratio (PSNR) values of 19.21 dB, 15.86 dB, and 13.62 dB are obtained for the training, validation, and test datasets, respectively. The average values of the structural similarity index (SSIM) are 0.83, 0.68, and 0.59, respectively, and are compared and discussed with the corresponding indicators of the other two models. The sensing system built using this method will show application potential in many fields that require accurate and real-time NLOS imaging, especially smart home security systems in room-scale scenes.

Plug-and-Play Algorithms for Dynamic Non-line-of-sight Imaging

Non-line-of-sight (NLOS) imaging has the ability to recover 3D images of scenes outside the direct line of sight, which is of growing interest for diverse applications. Despite the remarkable progress, NLOS imaging of dynamic objects is still challenging. It requires a large amount of multibounce photons for the reconstruction of single-frame data. To overcome this obstacle, we develop a computational framework for dynamic time-of-flight NLOS imaging based on plug-and-play (PnP) algorithms. By combining imaging forward model with the deep denoising network from the computer vision community, we show a 4 frames-per-second (fps) 3D NLOS video recovery (128 × 128 × 512) in post-processing. Our method leverages the temporal similarity among adjacent frames and incorporates sparse priors and frequency filtering. This enables higher-quality reconstructions for complex scenes. Extensive experiments are conducted to verify the superior performance of our proposed algorithm both through simulations and real data.

Non-Line-of-Sight Imaging and Vibrometry Using a Comb-Calibrated Coherent Sensor.

Non-line-of-sight (NLOS) imaging has the ability to reconstruct hidden objects, allowing a wide range of applications. Existing NLOS systems rely on pulsed lasers and time-resolved single-photon detectors to capture the information encoded in the time of flight of scattered photons. Despite remarkable advances, the pulsed time-of-flight LIDAR approach has limited temporal resolution and struggles to detect the frequency-associated information directly. Here, we propose and demonstrate the coherent scheme-frequency-modulated continuous wave calibrated by optical frequency comb-for high-resolution NLOS imaging, velocimetry, and vibrometry. Our comb-calibrated coherent sensor presents a system temporal resolution at subpicosecond and its superior signal-to-noise ratio permits NLOS imaging of complex scenes under strong ambient light. We show the capability of NLOS localization and 3D imaging at submillimeter scale and demonstrate NLOS vibrometry sensing at an accuracy of dozen Hertz. Our approach unlocks the coherent LIDAR techniques for widespread use in imaging science and optical sensing.

Enhancing the spatial resolution of time-of-flight based non-line-of-sight imaging via instrument response function deconvolution.

Non-line-of-sight (NLOS) imaging retrieves the hidden scenes by utilizing the signals indirectly reflected by the relay wall. Benefiting from the picosecond-level timing accuracy, time-correlated single photon counting (TCSPC) based NLOS imaging can achieve theoretical spatial resolutions up to millimeter level. However, in practical applications, the total temporal resolution (also known as total time jitter, TTJ) of most current TCSPC systems exceeds hundreds of picoseconds due to the combined effects of multiple electronic devices, which restricts the underlying spatial resolution of NLOS imaging. In this paper, an instrument response function deconvolution (IRF-DC) method is proposed to overcome the constraints of a TCSPC system's TTJ on the spatial resolution of NLOS imaging. Specifically, we model the transient measurements as Poisson convolution process with the normalized IRF as convolution kernel, and solve the inverse problem with iterative deconvolution algorithm, which significantly improves the spatial resolution of NLOS imaging after reconstruction. Numerical simulations show that the IRF-DC facilitates light-cone transform and frequency-wavenumber migration solver to achieve successful reconstruction even when the system's TTJ reaches 1200 ps, which is equivalent to what was previously possible when TTJ was about 200ps. In addition, the IRF-DC produces satisfactory reconstruction outcomes when the signal-to-noise ratio (SNR) is low. Furthermore, the effectiveness of the proposed method has also been experimentally verified. The proposed IRF-DC method is highly applicable and efficient, which may promote the development of high-resolution NLOS imaging.

Cohesive framework for non-line-of-sight imaging based on Dirac notation.

The non-line-of-sight (NLOS) imaging field encompasses both experimental and computational frameworks that focus on imaging elements that are out of the direct line-of-sight, for example, imaging elements that are around a corner. Current NLOS imaging methods offer a compromise between accuracy and reconstruction time as experimental setups have become more reliable, faster, and more accurate. However, all these imaging methods implement different assumptions and light transport models that are only valid under particular circumstances. This paper lays down the foundation for a cohesive theoretical framework which provides insights about the limitations and virtues of existing approaches in a rigorous mathematical manner. In particular, we adopt Dirac notation and concepts borrowed from quantum mechanics to define a set of simple equations that enable: i) the derivation of other NLOS imaging methods from such single equation (we provide examples of the three most used frameworks in NLOS imaging: back-propagation, phasor fields, and f-k migration); ii) the demonstration that the Rayleigh-Sommerfeld diffraction operator is the propagation operator for wave-based imaging methods; and iii) the demonstration that back-propagation and wave-based imaging formulations are equivalent since, as we show, propagation operators are unitary. We expect that our proposed framework will deepen our understanding of the NLOS field and expand its utility in practical cases by providing a cohesive intuition on how to image complex NLOS scenes independently of the underlying reconstruction method.

Miniaturized time-correlated single-photon counting module for time-of-flight non-line-of-sight imaging applications.

Single-photon time-of-flight (TOF) non-line-of-sight (NLOS) imaging enables the high-resolution reconstruction of objects outside the field of view. The compactness of TOF NLOS imaging systems, entailing the miniaturization of key components within such systems, is crucial for practical applications. Here, we present a miniaturized four-channel time-correlated single-photon counting module dedicated to TOF NLOS imaging applications. The module achieves excellent performance with a 10ps bin size and 27.4ps minimum root-mean-square time resolution. We present the results of the TOF NLOS imaging experiment using an InGaAs/InP single-photon detector and the time-correlated single-photon counting module and show that a 6.3cm lateral resolution and 2.3cm depth resolution can be achieved under the conditions of 5m imaging distance and 1ms pixel dwell time.

Towards a more accurate light transport model for non-line-of-sight imaging.

Non-line-of-sight (NLOS) imaging systems involve the measurement of an optical signal at a diffuse surface. A forward model encodes the physics of these measurements mathematically and can be inverted to generate a reconstruction of the hidden scene. Some existing NLOS imaging techniques rely on illuminating the diffuse surface and measuring the photon time of flight (ToF) of multi-bounce light paths. Alternatively, some methods depend on measuring high-frequency variations caused by shadows cast by occluders in the hidden scene. While forward models for ToF-NLOS and Shadow-NLOS have been developed separately, there has been limited work on unifying these two imaging modalities. Dove et al introduced a unified mathematical framework capable of modeling both imaging techniques [Opt. Express27, 18016 (2019)10.1364/OE.27.018016]. The authors utilize this general forward model, known as the two frequency spatial Wigner distribution (TFSWD), to discuss the implications of reconstruction resolution for combining the two modalities but only when the occluder geometry is known a priori. In this work, we develop a graphical representation of the TFSWD forward model and apply it to novel experimental setups with potential applications in NLOS imaging. Furthermore, we use this unified framework to explore the potential of combining these two imaging modalities in situations where the occluder geometry is not known in advance.

A Gradient-Gated SPAD Array for Non-Line-of-Sight Imaging

Time-resolved non-line-of-sight (NLOS) imaging based on single-photon avalanche diode (SPAD) detectors have demonstrated impressive results in recent years. To acquire adequate number of indirect photons from a hidden scene in the presence of overwhelming amount of early-arrival photons, a single-gated SPAD is widely employed to mitigate pile-up. However, additional prior knowledge of the hidden range and relay surface profile are required to preset the gating position and implement the reconstruction. With this work, we propose a gradient-gated technique to alleviate these shortcomings (pile-up and prior knowledge) in NLOS imaging with the development of a 6 × 6 SPAD array fabricated in a standard 55-nm Bipolar-CMOS-DMOS (BCD) technology. The SPAD sensor includes a delay-locked loop (DLL) block, a gating generator and a pixel array, which can be flexibly configured to free-running, single-gating and gradient-gating modes. The rise time of the gates is less than 450 ps (from 10% to 90%). In gradient-gating mode, the interval time between adjacent gates can be configured from 300 ps to 2 ns. In single-gating mode, the minimum gate window is measured below 5 ns. We demonstrated the sensor in a confocal NLOS imaging system that reconstructs hidden scenes with both retroreflective and diffuse objects. The presented scheme enables the development of calibration-free NLOS imaging modality for practical applications.

Physics to the Rescue: Deep Non-Line-of-Sight Reconstruction for High-Speed Imaging.

Computational approach to imaging around the corner, or non-line-of-sight (NLOS) imaging, is becoming a reality thanks to major advances in imaging hardware and reconstruction algorithms. A recent development towards practical NLOS imaging, Nam et al. [1] demonstrated a high-speed non-confocal imaging system that operates at 5Hz, 100x faster than the prior art. This enormous gain in acquisition rate, however, necessitates numerous approximations in light transport, breaking many existing NLOS reconstruction methods that assume an idealized image formation model. To bridge the gap, we present a novel deep model that incorporates the complementary physics priors of wave propagation and volume rendering into a neural network for high-quality and robust NLOS reconstruction. This orchestrated design regularizes the solution space by relaxing the image formation model, resulting in a deep model that generalizes well on real captures despite being exclusively trained on synthetic data. Further, we devise a unified learning framework that enables our model to be flexibly trained using diverse supervision signals, including target intensity images or even raw NLOS transient measurements. Once trained, our model renders both intensity and depth images at inference time in a single forward pass, capable of processing more than 5 captures per second on a high-end GPU. Through extensive qualitative and quantitative experiments, we show that our method outperforms prior physics and learning based approaches on both synthetic and real measurements. We anticipate that our method along with the fast capturing system will accelerate future development of NLOS imaging for real world applications that require high-speed imaging.

Isolating Signals in Passive Non-Line-of-Sight Imaging using Spectral Content.

In real-life passive non-line-of-sight (NLOS) imaging there is an overwhelming amount of undesired scattered radiance, called clutter, that impedes reconstruction of the desired NLOS scene. This paper explores using the spectral domain of the scattered light field to separate the desired scattered radiance from the clutter. We propose two techniques: The first separates the multispectral scattered radiance into a collection of objects each with their own uniform color. The objects which correspond to clutter can then be identified and removed based on how well they can be reconstructed using NLOS imaging algorithms. This technique requires very few priors and uses off-the-shelf algorithms. For the second technique, we derive and solve a convex optimization problem assuming we know the desired signal's spectral content. This method is quicker and can be performed with fewer spectral measurements. We demonstrate both techniques using realistic scenarios. In the presence of clutter that is 50 times stronger than the desired signal, the proposed reconstruction of the NLOS scene is 23 times more accurate than typical reconstructions and 5 times more accurate than using the leading clutter rejection method.

Single-shot non-line-of-sight imaging based on chromato-axial differential correlography

Non-line-of-sight (NLOS) imaging is a challenging task aimed at reconstructing objects outside the direct view of the observer. Nevertheless, traditional NLOS imaging methods typically rely on intricate and costly equipment to scan and sample the hidden object. These methods often suffer from restricted imaging resolution and require high system stability. Herein, we propose a single-shot high-resolution NLOS imaging method via chromato-axial differential correlography, which adopts low-cost continuous-wave lasers and a conventional camera. By leveraging the uncorrelated laser speckle patterns along the chromato-axis, this method can reconstruct hidden objects of diverse complexity using only one exposure measurement. The achieved background stability through single-shot acquisition, along with the inherent information redundancy in the chromato-axial differential speckles, enhances the robustness of the system against vibration and colored stain interference. This approach overcomes the limitations of conventional methods by simplifying the sampling process, improving system stability, and achieving enhanced imaging resolution using available equipment. This work serves as a valuable reference for the real-time development and practical implementation of NLOS imaging.

PI-NLOS: polarized infrared non-line-of-sight imaging.

Passive non-line-of-sight (NLOS) imaging is a promising technique to enhance visual perception for the occluded object hidden behind the wall. Here we present a data-driven NLOS imaging framework by using polarization cue and long-wavelength infrared (LWIR) images. We design a dual-channel input deep neural network to fuse the intensity features from polarized LWIR images and contour features from polarization degree images for NLOS scene reconstruction. To train the model, we create a polarized LWIR NLOS dataset which contains over ten thousand images. The paper demonstrates the passive NLOS imaging experiment in which the hidden people is approximate 6 meters away from the relay wall. It is an exciting finding that even the range is further than that in the prior works. The quantitative evaluation metric of PSNR and SSIM show that our method as an advance over state-of-the-art in passive NLOS imaging.

Non-line-of-sight imaging at infrared wavelengths using a superconducting nanowire single-photon detector

Non-line-of-sight (NLOS) imaging can visualize a remote object out of the direct line of sight and can potentially be used in endoscopy, unmanned vehicles, and robotic vision. In an NLOS imaging system, multiple diffusive reflections of light usually induce large optical attenuation, and therefore, a sensitive and efficient photodetector, or, their array, is required. Limited by the spectral sensitivity of the light sensors, up to now, most of the NLOS imaging experiments are performed in the visible bands, and a few at the near-infrared, 1550 nm. Here, to break this spectral limitation, we demonstrate a proof-of-principle NLOS imaging system using a fractal superconducting nanowire single-photon detector, which exhibits intrinsic single-photon sensitivity over an ultra-broad spectral range. We showcase NLOS imaging at 1560- and 1997-nm two wavelengths, both technologically important for specific applications. We develop a de-noising algorithm and combine it with the light-cone-transform algorithm to reconstruct the shape of the hidden objects with significantly enhanced signal-to-noise ratios. We believe that the joint advancement of the hardware and the algorithm presented in this paper could further expand the application spaces of the NLOS imaging systems.

Non-line-of-sight reconstruction via structure sparsity regularization.

Non-line-of-sight (NLOS) imaging allows for the imaging of objects around a corner, which enables potential applications in various fields, such as autonomous driving, robotic vision, medical imaging, security monitoring, etc. However, the quality of reconstruction is challenged by low signal-to-noise ratio (SNR) measurements. In this study, we present a regularization method, referred to as structure sparsity (SS) regularization, for denoising in NLOS reconstruction. By exploiting the prior knowledge of structure sparseness, we incorporate nuclear norm penalization into the cost function of the directional light-cone transform (DLCT) model for the NLOS imaging system. This incorporation effectively integrates the neighborhood information associated with the directional albedo, thereby facilitating the denoising process. Subsequently, the reconstruction is achieved by optimizing a directional albedo model with SS regularization using the fast iterative shrinkage-thresholding algorithm (FISTA). Notably, the robust reconstruction of occluded objects is observed. Through comprehensive evaluations conducted on both synthetic and experimental datasets, we demonstrate that the proposed approach yields high-quality reconstructions, surpassing the state-of-the-art reconstruction algorithms, especially in scenarios involving short exposure and low-SNR measurements.

Virtual Mirrors: Non-Line-of-Sight Imaging Beyond the Third Bounce

Non-line-of-sight (NLOS) imaging methods are capable of reconstructing complex scenes that are not visible to an observer using indirect illumination. However, they assume only third-bounce illumination, so they are currently limited to single-corner configurations, and present limited visibility when imaging surfaces at certain orientations. To reason about and tackle these limitations, we make the key observation that planar diffuse surfaces behave specularly at wavelengths used in the computational wave-based NLOS imaging domain. We call such surfaces virtual mirrors. We leverage this observation to expand the capabilities of NLOS imaging using illumination beyond the third bounce, addressing two problems: imaging single-corner objects at limited visibility angles, and imaging objects hidden behind two corners. To image objects at limited visibility angles, we first analyze the reflections of the known illuminated point on surfaces of the scene as an estimator of the position and orientation of objects with limited visibility. We then image those limited visibility objects by computationally building secondary apertures at other surfaces that observe the target object from a direct visibility perspective. Beyond single-corner NLOS imaging, we exploit the specular behavior of virtual mirrors to image objects hidden behind a second corner by imaging the space behind such virtual mirrors, where the mirror image of objects hidden around two corners is formed. No specular surfaces were involved in the making of this paper.

Non-line-of-sight imaging and location determination using deep learning

Non-line-of-sight (NLOS) imaging methods have been proposed and rapidly developed to address the challenge of detecting objects hidden from the direct line of sight. One critical issue in NLOS imaging is acquiring the spatial position of hidden objects. Active NLOS imaging methods solve this problem using complex designs and sophisticated equipment while the conventional passive NLOS imaging scheme, which does not have a controllable light source and time-gating detector, rarely captures the spatial information of objects. Using speckle patterns and the assistance of a deep learning method, we experimentally demonstrated that the passive NLOS imaging system could simultaneously visualize and locate objects hidden in corners like an active NLOS imaging system could. High-fidelity reconstructed images and high-accuracy location recognition were realized using the proposed network. The network could provide 99% recognition accuracy for the object's position with an axis spacing of 36 μm. These results were significant for NLOS imaging with high-accuracy positioning requirements.

Non-Line-of-Sight Detection Based on Neuromorphic Time-of-Flight Sensing

Non-line-of-sight (NLOS) detection and ranging aim to identify hidden objects by sensing indirect light reflections. Although numerous computational methods have been proposed for NLOS detection and imaging, the post-signal processing required by peripheral circuits remains complex. One possible solution for simplifying NLOS detection and ranging involves the use of neuromorphic devices, such as memristors, which have intrinsic resistive-switching capabilities and can store spatiotemporal information. In this study, we employed the memristive spike-timing-dependent plasticity learning rule to program the time-of-flight (ToF) depth information directly into a memristor medium. By coupling the transmitted signal from the source with the photocurrent from the target object into a single memristor unit, we were able to induce a tunable programming pulse based on the time interval between the two signals that were superimposed. Here, this neuromorphic ToF principle is employed to detect and range NLOS objects without requiring complex peripheral circuitry to process raw signals. We experimentally demonstrated the effectiveness of the neuromorphic ToF principle by integrating a HfO2 memristor and an avalanche photodiode to detect NLOS objects in multiple directions. This technology has potential applications in various fields, such as automotive navigation, machine learning, and biomedical engineering.