Photon-Efficient 3D Reconstruction with A Coarse-to-Fine Neural Network

Abstract

3D reconstruction from the sparse and noisy photon-efficient measurement Time-of-Arrival (ToA) cube is challenging because the effective echo signal occupies only a small part of the time channel and the rest of the time channel contains only noise. All learning-based photon-efficient 3D reconstruction methods extract features over the entire time channel of ToA. However, extracting features from the entire time channel causes the learned features to be mainly defined by noise accounting for most of the time channel, thereby reducing the 3D reconstruction accuracy. In this paper, we propose a Coarse-to-Fine neural network, where the coarse part eliminates the invalid noisy time bins and the fine part extracts features on the remaining time bins containing only effective echo signals. Specifically, due to locating the interval to which the effective echo signal belongs, non-local spatial-temporal features of the ToA cube must be captured. To this end, we propose a transformer-based Coarse-Interval-Localization-Network (CILN), which holds the global receptive field to aggregate features from long-distance time bins. Then, the located interval containing only the effective echo signal is cropped from the ToA cube and input to the proposed Fine-Maximum-Localization-Network (FMLN) to locate the maximum of the echo signal. Because the cropping operation destroys the distribution of the original signal, we propose the position encoding module to transmit the distribution change information to high dimensional feature space in the FMLN. Furthermore, we propose the temporal attention module to guide the FMLN to pay more attention to the useful signal. Compared with other methods that extract features over the entire time channel, the coarse-to-fine configuration of our method eliminates the time bins containing invalid noise through the coarse part and reduces the influence of the noise on the feature extraction of the fine part, and therefore the performance of the network on the reconstruction accuracy is improved. We conduct multiple experiments on the simulated data and real-world data, and the experimental results show that the proposed Coarse-to-Fine neural network can achieve state-of-the-art performance.