Non-line-of-sight Imaging Research Articles

Imaging operator in indirect imaging correlography.

Indirect imaging correlography (IIC) is a coherent imaging technique that provides access to the autocorrelation of the albedo of objects obscured from line-of-sight. This technique is used to recover sub-mm resolution images of obscured objects at large standoffs in non-line-of-sight (NLOS) imaging. However, predicting the exact resolving power of IIC in any given NLOS scene is complicated by the interplay between several factors, including object position and pose. This work puts forth a mathematical model for the imaging operator in IIC to accurately predict the images of objects in NLOS imaging scenes. Using the imaging operator, expressions for the spatial resolution as a function of scene parameters such as object position and pose are derived and validated experimentally. In addition, a self-supervised deep neural network framework to reconstruct images of objects from their autocorrelation is proposed. Using this framework, objects with ≈ 250 μ m features, located at 1 mt standoffs in an NLOS scene, are successfully reconstructed.

Non-line-of-sight imaging with arbitrary illumination and detection pattern

Non-line-of-sight (NLOS) imaging aims at reconstructing targets obscured from the direct line of sight. Existing NLOS imaging algorithms require dense measurements at regular grid points in a large area of the relay surface, which severely hinders their availability to variable relay scenarios in practical applications such as robotic vision, autonomous driving, rescue operations and remote sensing. In this work, we propose a Bayesian framework for NLOS imaging without specific requirements on the spatial pattern of illumination and detection points. By introducing virtual confocal signals, we design a confocal complemented signal-object collaborative regularization (CC-SOCR) algorithm for high-quality reconstructions. Our approach is capable of reconstructing both the albedo and surface normal of the hidden objects with fine details under general relay settings. Moreover, with a regular relay surface, coarse rather than dense measurements are enough for our approach such that the acquisition time can be reduced significantly. As demonstrated in multiple experiments, the proposed framework substantially extends the application range of NLOS imaging.

Compressive Non-Line-of-Sight Imaging with Deep Learning

In non-line-of-sight (NLOS) imaging, the spatial information of hidden targets is reconstructed from the time-of-light (TOF) of the multiple bounced signal photons. The need for NLOS imagers to perform extensive scanning in the transverse spatial dimensions constrains the imaging speed and reconstruction quality while limiting their applications on static scenes. Utilizing a photon TOF histogram with picosecond temporal resolution, we develop compressive non-line-of-sight imaging enabled by deep learning. Two-dimensional images ($32\ifmmode\times\else\texttimes\fi{}32$ pixels) of the NLOS targets can be reconstructed with superior reconstruction quality via a convolutional neural network (CNN), using significantly downscaled data ($8\ifmmode\times\else\texttimes\fi{}8$ scanning points) at a downsampling ratio of $6.25\mathrm{%}$ compared to the traditional methods. The CNN is end-to-end trained purely using simulated data but robust for image reconstruction with experiment data. Our results suggest that deep learning is effective for reducing the scanning points and total capture time towards scanningless NLOS imaging and videography.

Non-line-of-sight imaging algorithm based on Wiener filtering of mid-frequency

Non-line-of-sight (NLOS) imaging is an emerging technology for optically imaging the objects blocked beyond the detector's line of sight. The NLOS imaging based on light-cone transform and inverted method can be regarded as a deconvolution process. The traditional Wiener filtering deconvolution method uses the empirical values or the repeated attempts to obtain the power spectral density noise-to-signal ratio (PSDNSR) of the transient image: each hidden scene has a different PSDNSR for NLOS imaging, so the prior estimation is not appropriate and repeated attempts make it difficult to quickly find the optimal value. Therefore, in this work proposed is a method of estimating the PSDNSR by using the mid-frequency information of captured transient images for Wiener filtering to achieve NLOS imaging. In this method, the turning points between the mid-frequency domain and the high-frequency domain of the transient image amplitude spectrum are determined, and then the PSDNSR value is solved by analyzing the characteristics and relationship among the noise power spectra at the low, middle and high frequency. Experiments show that the PSDNSR estimated by NLOS imaging algorithm based on Wiener filtering of mid-frequency domain has a better reconstruction effect. Compared with other methods, the algorithm in this work can directly estimate PSDNSR in one step, without iterative operations, and the computational complexity is low, therebysimplifying the parameter adjustment steps of the Wiener filtering deconvolution NLOS imaging algorithm based on light-cone transform. Therefore the reconstruction efficiency can be improved on the premise of ensuring the reconstruction effect.

Passive non-line-of-sight imaging for moving targets with an event camera

Non-line-of-sight (NLOS) imaging is an emerging technique for detecting objects behind obstacles or around corners. Recent studies on passive NLOS mainly focus on steady-state measurement and reconstruction methods, which show limitations in recognition of moving targets. To the best of our knowledge, we propose a novel event-based passive NLOS imaging method. We acquire asynchronous event-based data which contains detailed dynamic information of the NLOS target, and efficiently ease the degradation of speckle caused by movement. Besides, we create the first event-based NLOS imaging dataset, NLOS-ES, and the event-based feature is extracted by time-surface representation. We compare the reconstructions through event-based data with frame-based data. The event-based method performs well on PSNR and LPIPS, which is 20% and 10% better than frame-based method, while the data volume takes only 2% of traditional method.

Long-Range Non-Line-of-Sight Imaging Based on Projected Images from Multiple Light Fields

Non-line-of-sight (NLOS) imaging technology has shown potential in several applications, such as intelligent driving, warfare and reconnaissance, medical diagnosis, and disaster rescue. However, most NLOS imaging systems are expensive and have a limited detection range, which hinders their utility in real-world scenarios. To address these limitations, we designed an NLOS imaging system, which is capable of long-range data acquisition. We also introduce an NLOS object imaging method based on deep learning, which makes use of long-range projected images from different light fields to reconstruct hidden objects. The method learns the mapping relationships of projected images and objects and corrects the image structure to suppress the generation of artifacts in order to improve the reconstruction quality. The results show that the proposed method produces fewer artifacts in reconstructions, which are close to human subjective perception. Furthermore, NLOS targets can be reconstructed even if the distance between the detection device and the intermediate surface exceeds 50 m.

Active mode single-pixel non-line-of-sight imaging system based on second-order correlation and diffraction

Non-line-of-sight (NLOS) imaging relies on capturing light that has been weakened due to multiple reflections and recovering the occluded scene with high quality is very challenging. To improve the quality of NLOS imaging, an active mode single-pixel NLOS imaging reconstruction framework based on the second-order correlation function and diffraction inverse operation is proposed. Under the collective effect of illumination pattern and single-pixel detector light intensity, the inverse imaging problem is first solved by a statistically weighted average, and then, an additional process is connected to reduce the influence of diffraction on reconstructed image quality based on the Fourier transform and inverse diffraction operation. The simulation and experimental results show that the proposed method with the Hadamard pattern and diffraction inverse operation, which achieved the SSIM value of 0.9286, is superior to the random pattern at a full sampling rate. The lens aperture size, pattern location, and the number of measurements have a significant effect on the system. Therefore, the proposed scheme has potential practical applications.

Deep Non-Line-of-Sight Imaging Using Echolocation.

Non-line-of-sight (NLOS) imaging is aimed at visualizing hidden scenes from an observer's (e.g., camera) viewpoint. Typically, hidden scenes are reconstructed using diffused signals that emit light sources using optical equipment and are reflected multiple times. Optical systems are commonly adopted in NLOS imaging because lasers can transport energy and focus light over long distances without loss. In contrast, we propose NLOS imaging using acoustic equipment inspired by echolocation. Existing acoustic NLOS is a computational method motivated by seismic imaging that analyzes the geometry of underground structures. However, this physical method is susceptible to noise and requires a clear signal, resulting in long data acquisition times. Therefore, we reduced the scan time by modifying the echoes to be collected simultaneously rather than sequentially. Then, we propose end-to-end deep-learning models to overcome the challenges of echoes interfering with each other. We designed three distinctive architectures: an encoder that extracts features by dividing multi-channel echoes into groups and merging them hierarchically, a generator that constructs an image of the hidden object, and a discriminator that compares the generated image with the ground-truth image. The proposed model successfully reconstructed the outline of the hidden objects.

Onsite Non-Line-of-Sight Imaging via Online Calibration

There has been increasing interest in deploying non-line-of-sight (NLOS) imaging systems for recovering objects hidden behind corners. Existing solutions need to calibrate the imaging system using auxiliary apparatus and additional detectors. We present an online calibration technique that directly decouples the transients, which are acquired by onsite scanning on a relay surface, into line-of-sight (LOS) and hidden components. We use the former to directly (re-)calibrate the system upon changes of scene <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$-$</tex-math></inline-formula> surface configurations, scannable regions, and sampling patterns, and the latter for hidden object recovery via spatial-, frequency-, or learning-based techniques. We also calculate a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Gamma</i> map from the LOS component to preview calibration effects for accurate transient measurements. The entire process of our calibration for 64 scanning points takes no more than 14 seconds on an Intel i7-6600H CPU. In particular, our technique avoids using auxiliary calibration tools such as mirrors or checkerboards and supports both uniform and non-uniform sampling in an onsite NLOS imaging system. Comprehensive experiments via calibration evaluation and NLOS reconstruction demonstrate the efficiency and effectiveness of our solution. Besides, we have made our data and code open-source on GitHub to the research community.

Non-line-of-sight imaging based on an untrained deep decoder network.

In recent years, low-cost high-quality non-line-of-sight (NLOS) imaging by a passive light source has been a significant research dimension. Here, we report a new, to the best of our knowledge, reconstruction method for the well-known "occluder-aided" NLOS imaging configuration based on an untrained deep decoder network. Using the interaction between the neural network and the physical forward model, the network weights can be automatically updated without the need for training data. Completion of the optimization process facilitates high-quality reconstructions of hidden scenes from photographs of a blank wall under high ambient light conditions. Simulations and experiments show the superior performance of the proposed method in terms of the details and the robustness of the reconstructed images. Our method will further promote the practical application of NLOS imaging in real scenes.

Non-line-of-sight imaging in the presence of scattering media using phasor fields.

Non-line-of-sight (NLOS) imaging aims to reconstruct partially or completely occluded scenes. Recent approaches have demonstrated high-quality reconstructions of complex scenes with arbitrary reflectance, occlusions, and significant multi-path effects. However, previous works focused on surface scattering only, which reduces the generality in more challenging scenarios such as scenes submerged in scattering media. In this work, we investigate current state-of-the-art NLOS imaging methods based on phasor fields to reconstruct scenes submerged in scattering media. We empirically analyze the capability of phasor fields in reconstructing complex synthetic scenes submerged in thick scattering media. We also apply the method to real scenes, showing that it performs similarly to recent diffuse optical tomography methods.

Fast Non-Line-Of-Sight Imaging with Two-Step Deep Remapping

Conventional imaging only records photons directly sent from the object to the detector, while non-line-of-sight (NLOS) imaging takes the indirect light into account. Most NLOS solutions employ a transient scanning process, followed by a physical based algorithm to reconstruct the NLOS scenes. However, the transient detection requires sophisticated apparatus, with long scanning time and low robustness to ambient environment, and the reconstruction algorithms are typically time-consuming and computationally expensive. Here we propose a new NLOS solution to address the above defects, with innovations on both equipment and algorithm. We apply inexpensive commercial Lidar for detection, with much higher scanning speed and better compatibility to real-world imaging. Our reconstruction framework is deep learning based, with a generative two-step remapping strategy to guarantee high reconstruction fidelity. The overall detection and reconstruction process allows for millisecond responses, with reconstruction precision of millimeter level. We have experimentally tested the proposed solution on both synthetic and real objects, and further demonstrated our method to be applicable to full-color NLOS imaging.

Fast non-line-of-sight imaging based on first photon event stamping.

Non-line-of-sight (NLOS) imaging enables people to see a hidden scene based on multiple interaction information between the object and the carrier. There have been numerous studies focusing on the physical modeling of photon scattering, but few have explored the detection process, which also plays a vital role. In this paper, we put forward a novel, to the best of our knowledge, detection methodology for NLOS imaging based on time-sequential first photon (TSFP) data. We verify the method with both synthetic and experimental data, showing a dramatic reduction in acquisition time cost compared with traditional methods for the same reconstruction quality. This work may contribute to real-time and photon-starved NLOS imaging for practical applications.

None-line-of-sight imaging enhanced with spatial multiplexing.

Non-line-of-sight (NLOS) imaging provides a fascinating way to see through obstacles. As one of the dominating NLOS imaging approaches, transient NLOS imaging uses ultrafast illumination and detection to sense hidden objects. Because ultrafast array detectors still face challenges in manufacture or cost, most existing transient NLOS imaging schemes use a point detector and therefore need a point-by-point scanning (PPS) process, rendering a relative low detection efficiency and long imaging time. In this work, we apply a passive mode single-pixel camera to implement spatial multiplexing detection (SMD) in NLOS imaging and achieve a higher efficiency of data acquisition. We analyze and demonstrate the superiority of SMD through both simulation and experiment. We also demonstrate a SMD scheme with compressed sensing (CS) strategy. A compression ratio as low as 18% is achieved. By utilizing SMD, we accomplish a boost of detection efficiency of up to 5 times compared with the traditional PPS mode. We believe that this SMD modality is certainly an important approach to prompt the development of NLOS imaging technologies.

FPGA Accelerator for Real-Time Non-Line-of-Sight Imaging

Non-line-of-sight (NLOS) imaging systems reconstruct hidden scenes using computational methods based on indirect light that diffusely reflected from relay walls. Due to the computation and memory requirements of reconstruction algorithms, real-time NLOS imaging for room-size scenes based on non-confocal data has long been challenging. This paper proposes a field programmable gate array (FPGA) accelerator for the recently proposed Rayleigh-Sommerfeld Diffraction (RSD)-based NLOS reconstruction method. In the proposed accelerator design, ring sampling and radius sampling techniques are proposed to reduce the memory requirements by reconstructing the RSD kernels with a set of kernel bases and ring sampling coefficients during the runtime. Based on that, a customized hardware architecture and the corresponding FPGA design for real-time RSD-based NLOS reconstruction is further proposed. Implementation results show that the proposed FPGA accelerator is capable of reconstructing NLOS scenes at 25 frames per second (FPS), running at a relatively slow clock frequency of 50 MHz. To the best knowledge of the authors, this is the first real-time enabled FPGA accelerator for room-size NLOS imaging with a resolution of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$128\times 128$ </tex-math></inline-formula> .

Computational adaptive optics for high-resolution non-line-of-sight imaging.

Non-line-of-sight (NLOS) imaging has aroused great interest during the past few years, by providing a unique solution for the observation of hidden objects behind obstructions or scattering media. As such, NLOS imaging may facilitate broad applications in autonomous driving, remote sensing, and medical diagnosis. However, existing NLOS frameworks suffer from severe degradation of resolution and signal-to-noise ratio (SNR) due to aberrations induced by scattering media and system misalignment, restricting its practical applications. This paper proposes a computational adaptive optics (CAO) method for NLOS imaging to correct optical aberrations in post-processing without the requirement of any hardware modifications. We demonstrate the effectiveness of CAO with a confocal NLOS imaging system in Terahertz (THz) band by imaging different samples behind occlusions for both low- and high-order aberrations. With appropriate metrics used for iterative CAO in post-processing, both the resolution and SNR can be increased by several times without reducing the data acquisition speed.

Computational framework for steady-state NLOS localization under changing ambient illumination conditions.

Non-line-of-sight (NLOS) imaging of hidden objects is a challenging yet vital task, facilitating important applications such as rescue operations, medical imaging, and autonomous driving. In this paper, we attempt to develop a computational steady-state NLOS localization framework that works accurately and robustly under various illumination conditions. For this purpose, we build a physical NLOS image acquisition hardware system and a corresponding virtual setup to obtain real-captured and simulated steady-state NLOS images under different ambient illuminations. Then, we utilize the captured NLOS images to train/fine-tune a multi-task convolutional neural network (CNN) architecture to perform simultaneous background illumination correction and NLOS object localization. Evaluation results on both stimulated and real-captured NLOS images demonstrate that the proposed method can effectively suppress severe disturbance caused by the variation of ambient light, significantly improving the accuracy and stability of steady-state NLOS localization using consumer-grade RGB cameras. The proposed method potentially paves the way to develop practical steady-state NLOS imaging solutions for around-the-clock and all-weather operations.

Passive Non-Line-of-Sight Imaging Using Optimal Transport

Passive non-line-of-sight (NLOS) imaging has drawn great attention in recent years. However, all existing methods are in common limited to simple hidden scenes, low-quality reconstruction, and small-scale datasets. In this paper, we propose NLOS-OT, a novel passive NLOS imaging framework based on manifold embedding and optimal transport, to reconstruct high-quality complicated hidden scenes. NLOS-OT converts the high-dimensional reconstruction task to a low-dimensional manifold mapping through optimal transport, alleviating the ill-posedness in passive NLOS imaging. Besides, we create the first large-scale passive NLOS imaging dataset, NLOS-Passive, which includes 50 groups and more than 3,200,000 images. NLOS-Passive collects target images with different distributions and their corresponding observed projections under various conditions, which can be used to evaluate the performance of passive NLOS imaging algorithms. It is shown that the proposed NLOS-OT framework achieves much better performance than the state-of-the-art methods on NLOS-Passive. We believe that the NLOS-OT framework together with the NLOS-Passive dataset is a big step and can inspire many ideas towards the development of learning-based passive NLOS imaging. Codes and dataset are publicly available (https://github.com/ruixv/NLOS-OT).

Single photon imaging and sensing of highly obscured objects around the corner

Non-line-of-sight (NLOS) optical imaging and sensing of objects imply new capabilities valuable to autonomous technology, machine vision, and other applications, in which case very few informative photons are buried in strong background counts. Here, we introduce a new approach to NLOS imaging and sensing using the picosecond-gated single photon detection generated by nonlinear frequency conversion. With exceptional signal isolation, this approach can reliably achieve imaging and position retrieval of obscured objects around the corner, in which case only 4 × 10−3 photons are needed to be detected per pulse for each pixel with high temporal resolution. Furthermore, the vibration frequencies of different objects can be resolved by analyzing the photon number fluctuation received within a ten-picosecond window, allowing NLOS acoustic sensing. Our results highlight the prospect of photon efficient NLOS imaging and sensing for real-world applications.

Non-line-of-sight imaging under white-light illumination: a two-step deep learning approach.

Non-line-of-sight (NLOS) imaging has received considerable attentions for its ability to recover occluded objects from an indirect view. Various NLOS imaging techniques have been demonstrated recently. Here, we propose a white-light NLOS imaging method that is equipped only with an ordinary camera, and not necessary to operate under active coherent illumination as in other existing NLOS systems. The central idea is to incorporate speckle correlation-based model into a deep neural network (DNN), and form a two-step DNN strategy that endeavors to learn the optimization of the scattered pattern autocorrelation and object image reconstruction, respectively. Optical experiments are carried out to demonstrate the proposed method.