Dense Depth Map Research Articles

EIL‐SLAM: Depth‐enhanced edge‐based infrared‐LiDAR SLAM

AbstractTraditional simultaneous localization and mapping (SLAM) approaches that utilize visible cameras or light detection and rangings (LiDARs) frequently fail in dusty, low‐textured, or completely dark environments. To address this problem, this study proposes a novel approach by tightly coupling perception data from a thermal infrared camera and a LiDAR based on the advantages of the former. However, applying a thermal infrared camera directly to existing SLAM frameworks is difficult because of the sensor differences. Thus, a new infrared‐visual odometry method is developed by utilizing edge points as features to ensure the robustness of the state estimation. Furthermore, an edge‐based infrared‐LiDAR SLAM framework is developed to generate a dense depth map for recovering visual scale and to provide real‐time pose estimation at the same time throughout the day. An infrared‐visual and LiDAR‐integrated place recognition method is also introduced to achieve robust loop closure. Finally, several experiments are performed to illustrate the effectiveness of the proposed approach.

CodeMapping: Real-Time Dense Mapping for Sparse SLAM using Compact Scene Representations

We propose a novel dense mapping framework for sparse visual SLAM systems which leverages a compact scene representation. State-of-the-art sparse visual SLAM systems provide accurate and reliable estimates of the camera trajectory and locations of landmarks. While these sparse maps are useful for localization, they cannot be used for other tasks such as obstacle avoidance or scene understanding. In this letter we propose a dense mapping framework to complement sparse visual SLAM systems which takes as input the camera poses, keyframes and sparse points produced by the SLAM system and predicts a dense depth image for every keyframe. We build on CodeSLAM [1] and use a variational autoencoder (VAE) which is conditioned on intensity, sparse depth and reprojection error images from sparse SLAM to predict an uncertainty-aware dense depth map. The use of a VAE then enables us to refine the dense depth images through multi-view optimization which improves the consistency of overlapping frames. Our mapper runs in a separate thread in parallel to the SLAM system in a loosely coupled manner. This flexible design allows for integration with arbitrary metric sparse SLAM systems without delaying the main SLAM process. Our dense mapper can be used not only for local mapping but also globally consistent dense 3D reconstruction through TSDF fusion. We demonstrate our system running with ORB-SLAM3 and show accurate dense depth estimation which could enable applications such as robotics and augmented reality.

Scanline Resolution-Invariant Depth Completion Using a Single Image and Sparse LiDAR Point Cloud

Most existing deep learning-based depth completion methods are only suitable for high (e.g. 64-scanline) resolution LiDAR measurements, and they usually fail to predict a reliable dense depth map with low resolution (4, 8, or 16-scanline) LiDAR. However, it is of great interest to reduce the number of LiDAR channels in many aspects (cost, weight of a device, power consumption). In this letter, we propose a new depth completion framework with various LiDAR scanline resolutions, which performs as well as methods built for 64-scanline resolution LiDAR inputs. For this, we define a consistency loss between the predictions from LiDAR measurements of different scanline resolutions. (i.e., 4, 8, 16, 32-scanline LiDAR measurements) Also, we design a fusion module to integrate features from different modalities. Experiments show our proposed method outperforms the current state-of-the-art depth completion methods for input LiDAR measurements of low scanline resolution and performs comparably to the methods(models) for input LiDAR measurements of 64-scanline resolution on the KITTI benchmark dataset.

SGTBN: Generating Dense Depth Maps From Single-Line LiDAR

Depth completion aims to generate a dense depth map from the sparse depth map and aligned RGB image. However, current depth completion methods use extremely expensive 64-line LiDAR(about $100,000) to obtain sparse depth maps, which will limit their application scenarios. Compared with the 64-line LiDAR, the single-line LiDAR is much less expensive and much more robust. Therefore, we propose a method to tackle the problem of single-line depth completion, in which we aim to generate a dense depth map from the single-line LiDAR info and the aligned RGB image. A single-line depth completion dataset is proposed based on the existing 64-line depth completion dataset(KITTI). A network called Semantic Guided Two-Branch Network(SGTBN) which contains global and local branches to extract and fuse global and local info is proposed for this task. A Semantic guided depth upsampling module is used in our network to make full use of the semantic info in RGB images. Except for the usual MSE loss, we add the virtual normal loss to increase the constraint of high-order 3D geometry in our network. Our network outperforms the state-of-the-art in the single-line depth completion task. Besides, compared with the monocular depth estimation, our method also has significant advantages in precision and model size.

Monocular Depth Estimation using Integrated Model with Multi-task Learning and Multi-stream

Scene understanding is a central problem in a field of computer vision. Depth estimation, in particular, is one of the important applications in scene understanding, robotics, and 3-D reconstruction. Estimating a dense depth map from a single image is receiving increased attention because a monocular camera is popular, small and suitable for a wide range of environments. In addition, both multi-task learning and multi-stream, which use unlabeled information, improve the monocular depth estimation efficiently. However, there are only a few networks optimized for both of them. Therefore, in this paper, we propose a monocular depth estimation task with a multi-task and multistream network architecture. Furthermore, the integrated network which we develop makes use of depth gradient information and can be applied to both supervised and unsupervised learning. In our experiments, we confirmed that our supervised learning architecture improves the accuracy of depth estimation by 0.13 m on average. Additionally, the experimental result on unsupervised learning found that it improved structure-from-motion performance.

Adaptive Multi-modal Fusion Instance Segmentation for CAEVs in Complex Conditions: Dataset, Framework and Verifications

Current works of environmental perception for connected autonomous electrified vehicles (CAEVs) mainly focus on the object detection task in good weather and illumination conditions, they often perform poorly in adverse scenarios and have a vague scene parsing ability. This paper aims to develop an end-to-end sharpening mixture of experts (SMoE) fusion framework to improve the robustness and accuracy of the perception systems for CAEVs in complex illumination and weather conditions. Three original contributions make our work distinctive from the existing relevant literature. The Complex KITTI dataset is introduced which consists of 7481 pairs of modified KITTI RGB images and the generated LiDAR dense depth maps, and this dataset is fine annotated in instance-level with the proposed semi-automatic annotation method. The SMoE fusion approach is devised to adaptively learn the robust kernels from complementary modalities. Comprehensive comparative experiments are implemented, and the results show that the proposed SMoE framework yield significant improvements over the other fusion techniques in adverse environmental conditions. This research proposes a SMoE fusion framework to improve the scene parsing ability of the perception systems for CAEVs in adverse conditions.

Depth Estimation for Monocular Image Based on Convolutional Neural Networks

Perceiving the three-dimensional structure of the surrounding environment and analyzing it for autonomous movement is an indispensable element for robots to operate in scenes. Recovering depth information and the three-dimensional spatial structure from monocular images is a basic mission of computer vision. For the objects in the image, there are many scenes that may produce it. This paper proposes to use a supervised end-to-end network to perform depth estimation without relying on any subsequent processing operations, such as probabilistic graphic models and other extra fine steps. This paper uses an encoder-decoder structure with feature pyramid to complete the prediction of dense depth maps. The encoder adopts ResNeXt-50 network to achieve main features from the original image. The feature pyramid structure can merge high and low level information with each other, and the feature information is not lost. The decoder utilizes the transposed convolutional and the convolutional layer to connect as an up-sampling structure to expand the resolution of the output. The structure adopted in this paper is applied to the indoor dataset NYU Depth v2 to obtain better prediction results than other methods. The experimental results show that on the NYU Depth v2 dataset, our method achieves the best results on 5 indicators and the sub-optimal results on 1 indicator.

가중 메디안 필터를 이용한 16채널 라이다의 깊이정보 업샘플링

Three-dimensional (3D) LiDAR is widely used because it can accurately measure 360° 3D depth. One of the most widely used sensors to this end is VLP-16, a 16-channel LiDAR manufactured by Velodyne. This 16-channel LiDAR provides dense 3D depth data in the horizontal direction. However, in the vertical direction, it provides only sparse depth data owing to the LiDAR architecture. We propose an upsampling method to obtain a dense 3D depth map from a 16-channel LiDAR by applying an edge-aware filter to the LiDAR data. A camera- LiDAR sensor system composed of a 16-channel LiDAR and six vision cameras is used for this purpose. First, the LiDAR depth data are projected onto the camera image coordinates to generate a 2.5D depth map. Next, bilinear interpolation and a weighted median filter are applied to upsample the original depth map. Compared with the original LiDAR data, we can generate an extremely dense depth map by using the proposed method. The results of quantitative analysis show that the proposed method can dramatically increase the number of 3D depth points and reduce the depth reconstruction error.

FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Depth Completion

Depth completion aims to recover a dense depth map from a sparse depth map with the corresponding color image as input. Recent approaches mainly formulate the depth completion as a one-stage end-to-end learning task, which outputs dense depth maps directly. However, the feature extraction and supervision in one-stage frameworks are insufficient, limiting the performance of these approaches. To address this problem, we propose a novel end-to-end residual learning framework, which formulates the depth completion as a two-stage learning task, i.e., a sparse-to-coarse stage and a coarse-to-fine stage. First, a coarse dense depth map is obtained by a simple CNN framework. Then, a refined depth map is further obtained using a residual learning strategy in the coarse-to-fine stage with coarse depth map and color image as input. Specially, in the coarse-to-fine stage, a channel shuffle extraction operation is utilized to extract more representative features from color image and coarse depth map, and an energy based fusion operation is exploited to effectively fuse these features obtained by channel shuffle operation, thus leading to more accurate and refined depth maps. We achieve SoTA performance in RMSE on KITTI benchmark. Extensive experiments on other datasets future demonstrate the superiority of our approach over current state-of-the-art depth completion approaches.

Self-Supervised Monocular Depth Learning in Low-Texture Areas

For the task of monocular depth estimation, self-supervised learning supervises training by calculating the pixel difference between the target image and the warped reference image, obtaining results comparable to those with full supervision. However, the problematic pixels in low-texture regions are ignored, since most researchers think that no pixels violate the assumption of camera motion, taking stereo pairs as the input in self-supervised learning, which leads to the optimization problem in these regions. To tackle this problem, we perform photometric loss using the lowest-level feature maps instead and implement first- and second-order smoothing to the depth, ensuring consistent gradients ring optimization. Given the shortcomings of ResNet as the backbone, we propose a new depth estimation network architecture to improve edge location accuracy and obtain clear outline information even in smoothed low-texture boundaries. To acquire more stable and reliable quantitative evaluation results, we introce a virtual data set in the self-supervised task because these have dense depth maps corresponding to pixel by pixel. We achieve performance that exceeds that of the prior methods on both the Eigen Splits of the KITTI and VKITTI2 data sets taking stereo pairs as the input.

Mean Shift Fusion Color Histogram Algorithm for Nonrigid Complex Target Tracking in Sports Video

We analyze and study the tracking of nonrigid complex targets of sports video based on mean shift fusion color histogram algorithm. A simple and controllable 3D template generation method based on monocular video sequences is constructed, which is used as a preprocessing stage of dynamic target 3D reconstruction algorithm to achieve the construction of templates for a variety of complex objects, such as human faces and human hands, broadening the use of the reconstruction method. This stage requires video sequences of rigid moving target objects or sets of target images taken from different angles as input. First, the standard rigid body method of Visuals is used to obtain the external camera parameters of the sequence frames as well as the sparse feature point reconstruction data, and the algorithm has high accuracy and robustness. Then, a dense depth map is computed for each input image frame by the Multi-View Stereo algorithm. The depth reconstruction with a too high resolution not only increases the processing time significantly but also generates more noise, so the resolution of the depth map is controlled by parameters. The multiple hypothesis target tracking algorithms are used to track multiple targets, while the chunking feature is used to solve the problem of mutual occlusion and adhesion between targets. After finishing the matching, the target and background models are updated online separately to ensure the validity of the target and background models. Our results of nonrigid complex target tracking by mean shift fusion color histogram algorithm for sports video improve the accuracy by about 8% compared to other studies. The proposed tracking method based on the mean shift algorithm and color histogram algorithm can not only estimate the position of the target effectively but also depict the shape of the target well, which solves the problem that the nonrigid targets in sports video have complicated shapes and are not easy to track. An example is given to demonstrate the effectiveness and adaptiveness of the applied method.

DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network

Depth Completion can produce a dense depth map from a sparse input and provide a more complete 3D description of the environment. Despite great progress made in depth completion, the sparsity of the input and low density of the ground truth still make this problem challenging. In this work, we propose DenseLiDAR, a novel real-time pseudo-depth guided depth completion neural network. We exploit dense pseudo-depth map obtained from simple morphological operations to guide the network in three aspects: (1) Constructing a residual structure for the output; (2) Rectifying the sparse input data; (3) Providing dense structural loss for training the network. Thanks to these novel designs, higher performance of the output could be achieved. In addition, two new metrics for better evaluating the quality of the predicted depth map are also presented. Extensive experiments on KITTI depth completion benchmark suggest that our model is able to achieve the state-of-the-art performance at the highest frame rate of 50Hz. The predicted dense depth is further evaluated by several downstream robotic perception or positioning tasks. For the task of 3D object detection, 3~5 percent performance gains on small objects categories are achieved on KITTI 3D object detection dataset. For RGB-D SLAM, higher accuracy on vehicle's trajectory is also obtained in KITTI Odometry dataset. These promising results not only verify the high quality of our depth prediction, but also demonstrate the potential of improving the related downstream tasks by using depth completion results.

An Adaptive Framework for Learning Unsupervised Depth Completion

We present a method to infer a dense depth map from a color image and associated sparse depth measurements. Our main contribution lies in the design of an annealing process for determining co-visibility (occlusions, disocclusions) and the degree of regularization to impose on the model. We show that regularization and co-visibility are related via the fitness (residual) of model to data and both can be unified into a single framework to improve the learning process. Our method is an adaptive weighting scheme that guides optimization by measuring the residual at each pixel location over each training step for (i) estimating a soft visibility mask and (ii) determining the amount of regularization. We demonstrate the effectiveness our method by applying it to several recent unsupervised depth completion methods and improving their performance on public benchmark datasets, without incurring additional trainable parameters or increase in inference time.

Superimposing height-controllable and animated flood surfaces into street-level photographs for risk communication

Extreme flood events frequently threaten coastal and river communities, and communicating the potential impacts of such forecasts to their populations is crucial to protect property and human life. However, traditional methods to warn residents of forecasted flood events are often ignored or not fully understood. Recent works have produced 3D visualizations of flooding to better capture viewers’ attentions but tend to be expensive, visually unrealistic, or incapable of parameterizing water height. Here we propose an efficient and scientifically-grounded approach to generate realistic images and animations of a flood at any height composited with a photograph taken at street level. Using vehicular LIDAR point cloud and color photo data, we employ a convolutional neural network to generate a dense depth map across an image. We use 3D modeling software to automatically generate and render a water surface, along with its own depth map, at the appropriate height and orientation. The depth maps are used to composite the photo with the rendered water surface to generate the final images, and this process can be repeated to generate videos of rising coastal floodwaters with animated waves within minutes. These visualizations are striking, and the overall framework can be supported by any particular image collection or depth map construction methodology, making this an affordable and achievable approach to flood risk communication.

Exploring Chromatic Aberration and Defocus Blur for Relative Depth Estimation From Monocular Hyperspectral Image.

This article investigates spectral chromatic and spatial defocus aberration in a monocular hyperspectral image (HSI) and proposes methods on how these cues can be utilized for relative depth estimation. The main aim of this work is to develop a framework by exploring intrinsic and extrinsic reflectance properties in HSI that can be useful for depth estimation. Depth estimation from a monocular image is a challenging task. An additional level of difficulty is added due to low resolution and noises in hyperspectral data. Our contribution to handling depth estimation in HSI is threefold. Firstly, we propose that change in focus across band images of HSI due to chromatic aberration and band-wise defocus blur can be integrated for depth estimation. Novel methods are developed to estimate sparse depth maps based on different integration models. Secondly, by adopting manifold learning, an effective objective function is developed to combine all sparse depth maps into a final optimized sparse depth map. Lastly, a new dense depth map generation approach is proposed, which extrapolate sparse depth cues by using material-based properties on graph Laplacian. Experimental results show that our methods successfully exploit HSI properties to generate depth cues. We also compare our method with state-of-the-art RGB image-based approaches, which shows that our methods produce better sparse and dense depth maps than those from the benchmark methods.

On the Confidence of Stereo Matching in a Deep-Learning Era: A Quantitative Evaluation.

Stereo matching is one of the most popular techniques to estimate dense depth maps by finding the disparity between matching pixels on two, synchronized and rectified images. Alongside with the development of more accurate algorithms, the research community focused on finding good strategies to estimate the reliability, i.e., the confidence, of estimated disparity maps. This information proves to be a powerful cue to naively find wrong matches as well as to improve the overall effectiveness of a variety of stereo algorithms according to different strategies. In this paper, we review more than ten years of developments in the field of confidence estimation for stereo matching. We extensively discuss and evaluate existing confidence measures and their variants, from hand-crafted ones to the most recent, state-of-the-art learning based methods. We study the different behaviors of each measure when applied to a pool of different stereo algorithms and, for the first time in literature, when paired with a state-of-the-art deep stereo network. Our experiments, carried out on five different standard datasets, provide a comprehensive overview of the field, highlighting in particular both strengths and limitations of learning-based strategies.

Learning Steering Kernels for Guided Depth Completion.

This paper addresses the guided depth completion task in which the goal is to predict a dense depth map given a guidance RGB image and sparse depth measurements. Recent advances on this problem nurture hopes that one day we can acquire accurate and dense depth at a very low cost. A major challenge of guided depth completion is to effectively make use of extremely sparse measurements, e.g., measurements covering less than 1% of the image pixels. In this paper, we propose a fully differentiable model that avoids convolving on sparse tensors by jointly learning depth interpolation and refinement. More specifically, we propose a differentiable kernel regression layer that interpolates the sparse depth measurements via learned kernels. We further refine the interpolated depth map using a residual depth refinement layer which leads to improved performance compared to learning absolute depth prediction using a vanilla network. We provide experimental evidence that our differentiable kernel regression layer not only enables end-to-end training from very sparse measurements using standard convolutional network architectures, but also leads to better depth interpolation results compared to existing heuristically motivated methods. We demonstrate that our method outperforms many state-of-the-art guided depth completion techniques on both NYUv2 and KITTI. We further show the generalization ability of our method with respect to the density and spatial statistics of the sparse depth measurements.

Adaptive Context-Aware Multi-Modal Network for Depth Completion

Depth completion aims to recover a dense depth map from the sparse depth data and the corresponding single RGB image. The observed pixels provide the significant guidance for the recovery of the unobserved pixels' depth. However, due to the sparsity of the depth data, the standard convolution operation, exploited by most of existing methods, is not effective to model the observed contexts with depth values. To address this issue, we propose to adopt the graph propagation to capture the observed spatial contexts. Specifically, we first construct multiple graphs at different scales from observed pixels. Since the graph structure varies from sample to sample, we then apply the attention mechanism on the propagation, which encourages the network to model the contextual information adaptively. Furthermore, considering the mutli-modality of input data, we exploit the graph propagation on the two modalities respectively to extract multi-modal representations. Finally, we introduce the symmetric gated fusion strategy to exploit the extracted multi-modal features effectively. The proposed strategy preserves the original information for one modality and also absorbs complementary information from the other through learning the adaptive gating weights. Our model, named Adaptive Context-Aware Multi-Modal Network (ACMNet), achieves the state-of-the-art performance on two benchmarks, i.e., KITTI and NYU-v2, and at the same time has fewer parameters than latest models. Our code is available at: https://github.com/sshan-zhao/ACMNet.

D-Net: A Generalised and Optimised Deep Network for Monocular Depth Estimation

Depth estimation is an essential component in computer vision systems for achieving 3D scene understanding. Efficient and accurate depth map estimation has numerous applications including self-driving vehicles and virtual reality tools. This paper presents a new deep network, called D-Net, for depth estimation from a single RGB image. The proposed network can be trained end-to-end, and its structure can be customised to meet different requirements in model size, speed, and prediction accuracy. Our approach gathers strong global and local contextual features at multiple resolutions, and then transfers these to high resolutions for clearer depth maps. For the encoder backbone, D-Net can utilise many state-of-the-art models including EfficientNet, HRNet and Swin Transformer to obtain dense depth maps. The proposed D-net is designed to have minimal parameters and reduced computational complexity. Extensive evaluations on the NYUv2 and KITTI benchmark datasets show that our model is highly accurate across multiple backbones, and it achieves state-of-the-art performance on both benchmarks when combined with the Swin Transformer and HRNets.

Feature Detection of Focused Plenoptic Camera Based on Central Projection Stereo Focal Stack

Fast and accurate feature extraction can lay a solid foundation for scene reconstruction and visual odometry. However, this has been rather a difficult problem for the focused plenoptic camera. In this paper, to the best of our knowledge, we first introduce an accurate and fast feature extraction algorithm based on central projection stereo focal stack (CPSFS). Specifically, we propose a refocusing algorithm that conforms to the central projection with regard to the center of main lens, which is more accurate than traditional one. On this basis, the feature is extracted on the CPSFS without calculating dense depth maps and total focus images. We verify the precision and efficiency of the proposed algorithm through simulated and real experiments, and give an example of scene reconstruction based on the proposed method. The experimental results show that our feature extraction algorithm is able to support the feature-based scene reconstruction via focused plenoptic camera.