Non-line Of Sight Reconstruction Research Articles

Physics to the Rescue: Deep Non-Line-of-Sight Reconstruction for High-Speed Imaging.

Computational approach to imaging around the corner, or non-line-of-sight (NLOS) imaging, is becoming a reality thanks to major advances in imaging hardware and reconstruction algorithms. A recent development towards practical NLOS imaging, Nam et al. [1] demonstrated a high-speed non-confocal imaging system that operates at 5Hz, 100x faster than the prior art. This enormous gain in acquisition rate, however, necessitates numerous approximations in light transport, breaking many existing NLOS reconstruction methods that assume an idealized image formation model. To bridge the gap, we present a novel deep model that incorporates the complementary physics priors of wave propagation and volume rendering into a neural network for high-quality and robust NLOS reconstruction. This orchestrated design regularizes the solution space by relaxing the image formation model, resulting in a deep model that generalizes well on real captures despite being exclusively trained on synthetic data. Further, we devise a unified learning framework that enables our model to be flexibly trained using diverse supervision signals, including target intensity images or even raw NLOS transient measurements. Once trained, our model renders both intensity and depth images at inference time in a single forward pass, capable of processing more than 5 captures per second on a high-end GPU. Through extensive qualitative and quantitative experiments, we show that our method outperforms prior physics and learning based approaches on both synthetic and real measurements. We anticipate that our method along with the fast capturing system will accelerate future development of NLOS imaging for real world applications that require high-speed imaging.

Non-line-of-sight reconstruction via structure sparsity regularization.

Non-line-of-sight (NLOS) imaging allows for the imaging of objects around a corner, which enables potential applications in various fields, such as autonomous driving, robotic vision, medical imaging, security monitoring, etc. However, the quality of reconstruction is challenged by low signal-to-noise ratio (SNR) measurements. In this study, we present a regularization method, referred to as structure sparsity (SS) regularization, for denoising in NLOS reconstruction. By exploiting the prior knowledge of structure sparseness, we incorporate nuclear norm penalization into the cost function of the directional light-cone transform (DLCT) model for the NLOS imaging system. This incorporation effectively integrates the neighborhood information associated with the directional albedo, thereby facilitating the denoising process. Subsequently, the reconstruction is achieved by optimizing a directional albedo model with SS regularization using the fast iterative shrinkage-thresholding algorithm (FISTA). Notably, the robust reconstruction of occluded objects is observed. Through comprehensive evaluations conducted on both synthetic and experimental datasets, we demonstrate that the proposed approach yields high-quality reconstructions, surpassing the state-of-the-art reconstruction algorithms, especially in scenarios involving short exposure and low-SNR measurements.

Ring and Radius Sampling Based Phasor Field Diffraction Algorithm for Non-Line-of-Sight Reconstruction.

Non-Line-of-Sight (NLOS) imaging reconstructs occluded scenes based on indirect diffuse reflections. The computational complexity and memory consumption of existing NLOS reconstruction algorithms make them challenging to be implemented in real-time. This paper presents a fast and memory-efficient phasor field-diffraction-based NLOS reconstruction algorithm. In the proposed algorithm, the radial property of the Rayleigh Sommerfeld diffraction (RSD) kernels along with the linear property of Fourier transform are utilized to reconstruct the Fourier domain representations of RSD kernels using a set of kernel bases. Moreover, memory consumption is further reduced by sampling the kernel bases in a radius direction and constructing them during the run-time. According to the analysis, the memory efficiency can be improved by as much as 220×. Experimental results show that compared with the original RSD algorithm, the reconstruction time of the proposed algorithm is significantly reduced with little impact on the final imaging quality.

Onsite Non-Line-of-Sight Imaging via Online Calibration

There has been increasing interest in deploying non-line-of-sight (NLOS) imaging systems for recovering objects hidden behind corners. Existing solutions need to calibrate the imaging system using auxiliary apparatus and additional detectors. We present an online calibration technique that directly decouples the transients, which are acquired by onsite scanning on a relay surface, into line-of-sight (LOS) and hidden components. We use the former to directly (re-)calibrate the system upon changes of scene <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$-$</tex-math></inline-formula> surface configurations, scannable regions, and sampling patterns, and the latter for hidden object recovery via spatial-, frequency-, or learning-based techniques. We also calculate a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Gamma</i> map from the LOS component to preview calibration effects for accurate transient measurements. The entire process of our calibration for 64 scanning points takes no more than 14 seconds on an Intel i7-6600H CPU. In particular, our technique avoids using auxiliary calibration tools such as mirrors or checkerboards and supports both uniform and non-uniform sampling in an onsite NLOS imaging system. Comprehensive experiments via calibration evaluation and NLOS reconstruction demonstrate the efficiency and effectiveness of our solution. Besides, we have made our data and code open-source on GitHub to the research community.

FPGA Accelerator for Real-Time Non-Line-of-Sight Imaging

Non-line-of-sight (NLOS) imaging systems reconstruct hidden scenes using computational methods based on indirect light that diffusely reflected from relay walls. Due to the computation and memory requirements of reconstruction algorithms, real-time NLOS imaging for room-size scenes based on non-confocal data has long been challenging. This paper proposes a field programmable gate array (FPGA) accelerator for the recently proposed Rayleigh-Sommerfeld Diffraction (RSD)-based NLOS reconstruction method. In the proposed accelerator design, ring sampling and radius sampling techniques are proposed to reduce the memory requirements by reconstructing the RSD kernels with a set of kernel bases and ring sampling coefficients during the runtime. Based on that, a customized hardware architecture and the corresponding FPGA design for real-time RSD-based NLOS reconstruction is further proposed. Implementation results show that the proposed FPGA accelerator is capable of reconstructing NLOS scenes at 25 frames per second (FPS), running at a relatively slow clock frequency of 50 MHz. To the best knowledge of the authors, this is the first real-time enabled FPGA accelerator for room-size NLOS imaging with a resolution of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$128\times 128$ </tex-math></inline-formula> .

Dynamic non-line-of-sight imaging system based on the optimization of point spread functions.

Non-line-of-sight (NLOS) imaging reveals hidden objects reflected from diffusing surfaces or behind scattering media. NLOS reconstruction is usually achieved by computational deconvolution of time-resolved transient data from a scanning single-photon avalanche diode (SPAD) detection system. However, using such a system requires a lengthy acquisition, impossible for capturing dynamic NLOS scenes. We propose to use a novel SPAD array and an optimization-based computational method to achieve NLOS reconstruction of 20 frames per second (fps). The imaging system's high efficiency drastically reduces the acquisition time for each frame. The forward projection optimization method robustly reconstructs NLOS scenes from low SNR data collected by the SPAD array. Experiments were conducted over a wide range of dynamic scenes in comparison with confocal and phase-field methods. Under the same exposure time, the proposed algorithm shows superior performances among state-of-the-art methods. To better analyze and validate our system, we also used simulated scenes to validate the advantages through quantitative benchmarks such as PSNR, SSIM and total variation analysis. Our system is anticipated to have the potential to achieve video-rate NLOS imaging.

Dataset for Benchmarking Time-Resolved Non-Line-of-Sight Imaging

Time-resolved imaging has made it possible to look around corners by exploiting information from indirect light bounces. While there have been successive improvements in the field since its conception, so far it has only been proven to work in very simple and controlled scenarios. We present a public dataset of synthetic time-resolved Non-Line-of-Sight (NLOS) scenes with varied complexity aimed at benchmarking reconstructions. It includes scenes that are common in the real world but remain a challenge for NLOS reconstruction methods due to the ambiguous nature of higher-order diffuse bounces naturally occurring in them. With over 300 reconstructible scenes, the dataset contains an order of magnitude more scenes than what is available currently. The final objective of the dataset it to boost NLOS research to take it closer to its real-world applications.

Richardson-Lucy deconvolution of time histograms for high-resolution non-line-of-sight imaging based on a back-projection method.

Non-line-of-sight (NLoS) imaging is a newly developed technique with widespread applications in recent years. At present, it is difficult for NLoS imaging to achieve a high-resolution reconstruction with an existing back-projection method. We propose a novel approach based on the Richardson-Lucy deconvolution of time histograms for high-resolution NLoS imaging. In our approach, the backscattering photons from a line-of-sight scene, often regarded as adverse factors, are used favorably as priors for the deconvolution of time histograms. All deconvoluted time histograms are back projected into the space to form a high-resolution NLoS reconstruction. The simulated and experimental results with the approach demonstrated that the reconstructions of NLoS objects are indeed with high resolution, even under a relatively high noise level.

Confocal non-line-of-sight imaging based on the light-cone transform.

How to image objects that are hidden from a camera's view is a problem of fundamental importance to many fields of research, with applications in robotic vision, defence, remote sensing, medical imaging and autonomous vehicles. Non-line-of-sight (NLOS) imaging at macroscopic scales has been demonstrated by scanning a visible surface with a pulsed laser and a time-resolved detector. Whereas light detection and ranging (LIDAR) systems use such measurements to recover the shape of visible objects from direct reflections, NLOS imaging reconstructs the shape and albedo of hidden objects from multiply scattered light. Despite recent advances, NLOS imaging has remained impractical owing to the prohibitive memory and processing requirements of existing reconstruction algorithms, and the extremely weak signal of multiply scattered light. Here we show that a confocal scanning procedure can address these challenges by facilitating the derivation of the light-cone transform to solve the NLOS reconstruction problem. This method requires much smaller computational and memory resources than previous reconstruction methods do and images hidden objects at unprecedented resolution. Confocal scanning also provides a sizeable increase in signal and range when imaging retroreflective objects. We quantify the resolution bounds of NLOS imaging, demonstrate its potential for real-time tracking and derive efficient algorithms that incorporate image priors and a physically accurate noise model. Additionally, we describe successful outdoor experiments of NLOS imaging under indirect sunlight.

Fast back-projection for non-line of sight reconstruction.

Recent works have demonstrated non-line of sight (NLOS) reconstruction by using the time-resolved signal from multiply scattered light. These works combine ultrafast imaging systems with computation, which back-projects the recorded space-time signal to build a probabilistic map of the hidden geometry. Unfortunately, this computation is slow, becoming a bottleneck as the imaging technology improves. In this work, we propose a new back-projection technique for NLOS reconstruction, which is up to a thousand times faster than previous work, with almost no quality loss. We base on the observation that the hidden geometry probability map can be built as the intersection of the three-bounce space-time manifolds defined by the light illuminating the hidden geometry and the visible point receiving the scattered light from such hidden geometry. This allows us to pose the reconstruction of the hidden geometry as the voxelization of these space-time manifolds, which has lower theoretic complexity and is easily implementable in the GPU. We demonstrate the efficiency and quality of our technique compared against previous methods in both captured and synthetic data.