Depth Map Estimation Research Articles

Self-Supervised Depth and Ego-Motion Estimation for Monocular Thermal Video Using Multi-Spectral Consistency Loss

A thermal camera can robustly capture thermal radiation images under harsh light conditions such as night scenes, tunnels, and disaster scenarios. However, despite this advantage, neither depth nor ego-motion estimation research for the thermal camera have not been actively explored so far. In this paper, we propose a self-supervised learning method for depth and ego-motion estimation from thermal images. The proposed method exploits multi-spectral consistency that consists of temperature and photometric consistency loss. The temperature consistency loss provides a fundamental self-supervisory signal by reconstructing clipped and colorized thermal images. Additionally, we design a differentiable forward warping module that can transform the coordinate system of the estimated depth map and relative pose from thermal camera to visible camera. Based on the proposed module, the photometric consistency loss can provide complementary self-supervision to networks. Networks trained with the proposed method robustly estimate the depth and pose from monocular thermal video under low-light and even zero-light conditions. To the best of our knowledge, this is the first work to simultaneously estimate both depth and ego-motion from monocular thermal video in a self-supervised manner.

SelfVIO: Self-supervised deep monocular Visual–Inertial Odometry and depth estimation

In the last decade, numerous supervised deep learning approaches have been proposed for visual–inertial odometry (VIO) and depth map estimation, which require large amounts of labelled data. To overcome the data limitation, self-supervised learning has emerged as a promising alternative that exploits constraints such as geometric and photometric consistency in the scene. In this study, we present a novel self-supervised deep learning-based VIO and depth map recovery approach (SelfVIO) using adversarial training and self-adaptive visual–inertial sensor fusion. SelfVIO learns the joint estimation of 6 degrees-of-freedom (6-DoF) ego-motion and a depth map of the scene from unlabelled monocular RGB image sequences and inertial measurement unit (IMU) readings. The proposed approach is able to perform VIO without requiring IMU intrinsic parameters and/or extrinsic calibration between IMU and the camera. We provide comprehensive quantitative and qualitative evaluations of the proposed framework and compare its performance with state-of-the-art VIO, VO, and visual simultaneous localization and mapping (VSLAM) approaches on the KITTI, EuRoC and Cityscapes datasets. Detailed comparisons prove that SelfVIO outperforms state-of-the-art VIO approaches in terms of pose estimation and depth recovery, making it a promising approach among existing methods in the literature.

Noise-Resilient Depth Estimation for Light Field Images Using Focal Stack and FFT Analysis.

Depth estimation for light field images is essential for applications such as light field image compression, reconstructing perspective views and 3D reconstruction. Previous depth map estimation approaches do not capture sharp transitions around object boundaries due to occlusions, making many of the current approaches unreliable at depth discontinuities. This is especially the case for light field images because the pixels do not exhibit photo-consistency in the presence of occlusions. In this paper, we propose an algorithm to estimate the depth map for light field images using depth from defocus. Our approach uses a small patch size of pixels in each focal stack image for comparing defocus cues, allowing the algorithm to generate sharper depth boundaries. Then, in contrast to existing approaches that use defocus cues for depth estimation, we use frequency domain analysis image similarity checking to generate the depth map. Processing in the frequency domain reduces the individual pixel errors that occur while directly comparing RGB images, making the algorithm more resilient to noise. The algorithm has been evaluated on both a synthetic image dataset and real-world images in the JPEG dataset. Experimental results demonstrate that our proposed algorithm outperforms state-of-the-art depth estimation techniques for light field images, particularly in case of noisy images.

A cascaded CNN-based method for monocular vision robotic grasping

PurposeScattered parts are laid randomly during the manufacturing process and have difficulty to recognize and manipulate. This study aims to complete the grasp of the scattered parts by a manipulator with a camera and learning method.Design/methodology/approachIn this paper, a cascaded convolutional neural network (CNN) method for robotic grasping based on monocular vision and small data set of scattered parts is proposed. This method can be divided into three steps: object detection, monocular depth estimation and keypoint estimation. In the first stage, an object detection network is improved to effectively locate the candidate parts. Then, it contains a neural network structure and corresponding training method to learn and reason high-resolution input images to obtain depth estimation. The keypoint estimation in the third step is expressed as a cumulative form of multi-scale prediction from a network to use an red green blue depth (RGBD) map that is acquired from the object detection and depth map estimation. Finally, a grasping strategy is studied to achieve successful and continuous grasping. In the experiments, different workpieces are used to validate the proposed method. The best grasping success rate is more than 80%.FindingsBy using the CNN-based method to extract the key points of the scattered parts and calculating the possibility of grasp, the successful rate is increased.Practical implicationsThis method and robotic systems can be used in picking and placing of most industrial automatic manufacturing or assembly processes.Originality/valueUnlike standard parts, scattered parts are randomly laid and have difficulty recognizing and grasping for the robot. This study uses a cascaded CNN network to extract the keypoints of the scattered parts, which are also labeled with the possibility of successful grasping. Experiments are conducted to demonstrate the grasping of those scattered parts.

Fusion of sparse LIDAR and uncertainty-based 3D vision for accurate depth estimation with Bayesian Kriging

Accurate 3D perception is vital for the localization of mobile robots and autonomous vehicles. Sparse 3D LIDAR scanners and 3D vision sensors (stereo and RGB-D cameras) are commonly used environment perception sensors. The former could provide accurate but sparse-range data, while the latter is dense but full of uncertainty. Since the acquired depth map by each sensor alone could not accurately perceive complex surrounding environment, an innovative probabilistic fusion model based on Bayesian Kriging (BK) is proposed for dense high-precision depth map estimation, to fully utilize the complementary characteristics of the two distinct sensors. Combing the geometric piece-wise planar hypotheses of the scene, the noisy disparity map from the 3D vision sensor is segmented into multiple small planes. In each segmented plane, the BK-based fusion estimator is founded to predict depth for each pixel with sparse scattered LIDAR points. In particular, we focus on analyzing the spatial variation prior and correlation between these two types of data, based on regionalized variable theory. Experiments on various scenes of KITTI stereo datasets are carried out to demonstrate the effectiveness of our algorithm, and validate the superiority compared with existing fusion methods.

Detection of Anomalous Objects on the Soil by Estimating Depth Map from RGB Image

Detection of Anomalous Objects on the Soil by Estimating Depth Map from RGB Image

Adaptive Illumination Based Depth Sensing Using Deep Superpixel and Soft Sampling Approximation

Dense depth map capture is challenging in existing active sparse illumination based depth acquisition techniques, such as LiDAR. Various techniques have been proposed to estimate a dense depth map based on fusion of the sparse depth map measurement with the RGB image. Recent advances in hardware enable adaptive depth measurements resulting in further improvement of the dense depth map estimation. In this paper, we study the topic of estimating dense depth from depth sampling. The adaptive sparse depth sampling network is jointly trained with a fusion network of an RGB image and sparse depth, to generate optimal adaptive sampling masks. Deep learning based superpixel sampling and soft sampling approximation are applied. We show that such adaptive sampling masks can generalize well to many RGB and sparse depth fusion algorithms under a variety of sampling rates (as low as 0.0625%). The proposed adaptive sampling method is fully differentiable and flexible to be trained end-to-end with upstream perception algorithms.

Multi-Scale Geometric Consistency Guided and Planar Prior Assisted Multi-View Stereo.

In this paper, we propose some efficient multi-view stereo methods for accurate and complete depth map estimation. We first present our basic methods with Adaptive Checkerboard sampling and Multi-Hypothesis joint view selection (ACMH & ACMH+). Based on our basic models, we develop two frameworks to deal with the depth estimation of ambiguous regions (especially low-textured areas) from two different perspectives: multi-scale information fusion and planar geometric clue assistance. For the former one, we propose a multi-scale geometric consistency guidance framework (ACMM) to obtain the reliable depth estimates for low-textured areas at coarser scales and guarantee that they can be propagated to finer scales. For the latter one, we propose a planar prior assisted framework (ACMP). We utilize a probabilistic graphical model to contribute a novel multi-view aggregated matching cost. At last, by taking advantage of the above frameworks, we further design a multi-scale geometric consistency guided and planar prior assisted multi-view stereo (ACMMP). This greatly enhances the discrimination of ambiguous regions and helps their depth sensing. Experiments on extensive datasets show our methods achieve state-of-the-art performance, recovering the depth estimation not only in low-textured areas but also in details. Related codes are available at https://github.com/GhiXu.

Accurate Analysis of the Spatial Pattern of Reflected Light and Surface Orientations Based on Color Illumination

3D Recovery approaches require a variety of clues to obtain shape information. The shape from shading (SFS) method uses shading information in images to estimate depth maps. Although shading contains detailed information, it causes some well-known ambiguities such as convex-concave ambiguity. In this study, a system installation, using red, green, and blue illumination, and an algorithm, processing reflections on the surface, were proposed for the accurate analysis of surface orientations, and ambiguity problems. Surface orientations, erroneously predicted by six different methods, were improved by implementing the proposed system. Consequently, the correct orientation of the surface points was determined by removing the ambiguities in images taken without considering the location of illumination, and all the tested methods provided successful results using the proposed system.

Uncertainty-Aware Self-Improving Framework for Depth Estimation

In recent years, self-supervised paradigms for depth estimation have drawn lots of attention from the community. Promising as they are, in order to achieve wider application and better performance, reasoning about the uncertainty of the estimated depth maps is in great demand. In this paper, we propose a novel uncertainty-aware self-improving framework for self-supervised depth estimation, which could model and exploit uncertainty to boost depth accuracy without any external supervision. In detail, our framework includes two DepthNets–a teacher network and a student network. The teacher network is pre-trained in a self-supervised manner and the predicted depth is leveraged as pseudo-ground-truth to supervise the learning of the student network, leading to better estimation accuracy and the ability of modeling depth uncertainty. Afterward, the estimated uncertainty would be introduced back into the teacher network as the mask, preventing erroneous regions from contributing to network training and depth estimation. In this way, a better pseudo-ground-truth from the teacher network will be used for the next iteration of the student network training. We evaluate our proposed framework on the widely used KITTI dataset and TUM RGB-D dataset with both quantitative and qualitative experiments. The experimental results show that our framework achieves quite satisfactory performance among the self-supervised depth estimation methods. Extensive experiments which combine the student network and feature-based monocular SLAM system (i.e ORB-SLAM3) further demonstrate the benefits of the introduced depth uncertainty.

Dense Depth-Map Estimation Based on Fusion of Event Camera and Sparse LiDAR

Depth-map estimation reflects the geometry of the visible surface in the environment directly and plays an important role in perception and decision for intelligent robots. However, sparse LiDAR only provides low-resolution depth information, which is a huge challenge for accurate sensing algorithms. To address this problem, this article proposes a novel fusion framework to generate dense depth-map based on event camera and sparse LiDAR. The approach uses the geometric information provided by the point cloud as prior knowledge and clusters point cloud data by an improved density clustering algorithm. Combined with the 3-D surface model of each cluster, the approach can provide 3-D reconstructions of the coordinate points of events and further obtain the dense-depth map by depth expansion and hole filling. Finally, we deploy our approach in MVSEC datasets and real-world applications. Experimental results show that, compared with other approaches, our approach can obtain more accurate depth information.

Single Image Depth Map Estimation for Improving Posture Recognition

Image-based posture recognition is a very challenging problem since it is difficult to acquire rich 3D information from the posture in color image. To address this issue, we present a novel and unified framework for human posture recognition, applying single image depth map estimation from color images. The proposed method includes two stages. The first stage estimates the depth map from the single-color image by an improved Pix2Pix generation module. The generation module is equipped with a hybrid loss function that captures the high-level features and recovers the sharp depth discontinuities, thus improving the depth estimation results. The second stage (the recognition stage) improves the color image-based recognition performance by incorporating the estimated depth map. Thereby, a two-stream CNN architecture that separately processes the color image and its estimated depth image is developed for robust posture recognition. To verify its effectiveness, we first test the proposed method on a novel pose dataset, which contains 13800 samples of paired color-and-depth of 6 subjects with 15 poses. The dataset used in this work is been created and released, is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://media.ritsumei.ac.jp/iipl/database/pose/</uri> . Extensive experiments are also performed on the public OUHANDS hand gesture dataset. Experiments demonstrate that the proposed method achieves superior performance on both human pose and hand gesture recognition tasks.

DP-MVS: Detail Preserving Multi-View Surface Reconstruction of Large-Scale Scenes

This paper presents an accurate and robust dense 3D reconstruction system for detail preserving surface modeling of large-scale scenes from multi-view images, which we named DP-MVS. Our system performs high-quality large-scale dense reconstruction, which preserves geometric details for thin structures, especially for linear objects. Our framework begins with a sparse reconstruction carried out by an incremental Structure-from-Motion. Based on the reconstructed sparse map, a novel detail preserving PatchMatch approach is applied for depth estimation of each image view. The estimated depth maps of multiple views are then fused to a dense point cloud in a memory-efficient way, followed by a detail-aware surface meshing method to extract the final surface mesh of the captured scene. Experiments on ETH3D benchmark show that the proposed method outperforms other state-of-the-art methods on F1-score, with the running time more than 4 times faster. More experiments on large-scale photo collections demonstrate the effectiveness of the proposed framework for large-scale scene reconstruction in terms of accuracy, completeness, memory saving, and time efficiency.

Computational Large Field-of-View RGB-D Integral Imaging System.

The integral imaging system has received considerable research attention because it can be applied to real-time three-dimensional image displays with a continuous view angle without supplementary devices. Most previous approaches place a physical micro-lens array in front of the image, where each lens looks different depending on the viewing angle. A computational integral imaging system with a virtual micro-lens arrays has been proposed in order to provide flexibility for users to change micro-lens arrays and focal length while reducing distortions due to physical mismatches with the lens arrays. However, computational integral imaging methods only represent part of the whole image because the size of virtual lens arrays is much smaller than the given large-scale images when dealing with large-scale images. As a result, the previous approaches produce sub-aperture images with a small field of view and need additional devices for depth information to apply to integral imaging pickup systems. In this paper, we present a single image-based computational RGB-D integral imaging pickup system for a large field of view in real time. The proposed system comprises three steps: deep learning-based automatic depth map estimation from an RGB input image without the help of an additional device, a hierarchical integral imaging system for a large field of view in real time, and post-processing for optimized visualization of the failed pickup area using an inpainting method. Quantitative and qualitative experimental results verify the proposed approach’s robustness.

EfficientNet-B0 Based Monocular Dense-Depth Map Estimation

Monocular depth estimation is a hot research topic in autonomous car driving. Deep convolution neural networks (DCNN) comprising encoder and decoder with transfer learning are exploited in the proposed work for monocular depth map estimation of two-dimensional images. Extracted CNN features from initial stages are later upsampled using a sequence of Bilinear UpSampling and convolution layers to reconstruct the depth map. The encoder forms the feature extraction part, and the decoder forms the image reconstruction part. EfficientNetB0, a new architecture is used with pretrained weights as encoder. It is a revolutionary architecture with smaller model parameters yet achieving higher efficiencies than the architectures of state-of-the-art, pretrained networks. EfficientNet-B0 is compared with two other pretrained networks, the DenseNet-121 and ResNet50 models. Each of these three models are used in encoding stage for features extraction followed by bilinear method of UpSampling in the decoder. The Monocular image is an ill-posed problem and is thus considered as a regression problem. So the metrics used in the proposed work are F1-score, Jaccard score and Mean Actual Error (MAE) etc., between the original and the reconstructed image. The results convey that EfficientNet-B0 outperforms in validation loss, F1-score and Jaccard score compared to DenseNet-121 and ResNet-50 models.

GAC3D: improving monocular 3D object detection with ground-guide model and adaptive convolution.

Monocular 3D object detection has recently become prevalent in autonomous driving and navigation applications due to its cost-efficiency and easy-to-embed to existent vehicles. The most challenging task in monocular vision is to estimate a reliable object’s location cause of the lack of depth information in RGB images. Many methods tackle this ill-posed problem by directly regressing the object’s depth or take the depth map as a supplement input to enhance the model’s results. However, the performance relies heavily on the estimated depth map quality, which is bias to the training data. In this work, we propose depth-adaptive convolution to replace the traditional 2D convolution to deal with the divergent context of the image’s features. This lead to significant improvement in both training convergence and testing accuracy. Second, we propose a ground plane model that utilizes geometric constraints in the pose estimation process. With the new method, named GAC3D, we achieve better detection results. We demonstrate our approach on the KITTI 3D Object Detection benchmark, which outperforms existing monocular methods.

The Synthesis of Unpaired Underwater Images for Monocular Underwater Depth Prediction

Underwater depth prediction plays an important role in underwater vision research. Because of the complex underwater environment, it is extremely difficult and expensive to obtain underwater datasets with reliable depth annotation. Thus, underwater depth map estimation with a data-driven manner is still a challenging task. To tackle this problem, we propose an end-to-end system including two different modules for underwater image synthesis and underwater depth map estimation, respectively. The former module aims to translate the hazy in-air RGB-D images to multi-style realistic synthetic underwater images while retaining the objects and the structural information of the input images. Then we construct a semi-real RGB-D underwater dataset using the synthesized underwater images and the original corresponding depth maps. We conduct supervised learning to perform depth estimation through the pseudo paired underwater RGB-D images. Comprehensive experiments have demonstrated that the proposed method can generate multiple realistic underwater images with high fidelity, which can be applied to enhance the performance of monocular underwater image depth estimation. Furthermore, the trained depth estimation model can be applied to real underwater image depth map estimation. We will release our codes and experimental setting in https://github.com/ZHAOQIII/UW_depth.

Underwater image restoration via depth map and illumination estimation based on a single image.

For the enhancement process of underwater images taken in various water types, previous methods employ the simple image formation model, thus obtaining poor restoration results. Recently, a revised underwater image formation model (i.e., the Akkaynak-Treibitz model) has shown better robustness in underwater image restoration, but has drawn little attention due to its complexity. Herein, we develop a dehazing method utilizing the revised model, which depends on the scene depth map and a color correction method to eliminate color distortion. Specifically, we first design an underwater image depth estimation method to create the depth map. Subsequently, according to the depth value of each pixel, the backscatter is estimated and removed by the channel based on the revised model. Furthermore, we propose a color correction approach to adjust the global color distribution of the image automatically. Our method only uses a single underwater image as input to eliminate lightwave absorption and scattering influence. Compared with state-of-the-art methods, both subjective and objective experimental results show that our approach can be applied to various real-world underwater scenes and has better contrast and color.

Depth completion towards different sensor configurations via relative depth map estimation and scale recovery

Depth completion, which combines additional sparse depth information from the range sensors, substantially improves the accuracy of monocular depth estimation, especially using the deep-learning-based methods. However, these methods can hardly produce satisfactory depth results when the sensor configuration changes at test time, which is important for real-world applications. In this paper, the problem is tackled by our proposed novel two-stage mechanism, which decomposes depth completion into two subtasks, namely relative depth map estimation and scale recovery. The relative depth map is first estimated from a single color image with our designed scale-invariant loss function. Then the scale map is recovered with the additional sparse depth. Experiments on different densities and patterns of the sparse depth input show that our model always produces satisfactory depth results. Besides, our approach achieves state-of-the-art performance on the indoor NYUv2 dataset and performs competitively on the outdoor KITTI dataset, demonstrating the effectiveness of our method.

Scene-Specific Dark Channel Prior for Single Image Fog Removal

Extensive presence of fog in outdoor images severely alters the scene appearance and hence reduces the visibility. Image processing based defogging algorithms are used to restore the details and colour in a single foggy image. Performances of the previous defogging approaches are considerably low since they fail to consider the image-specific cues. In this paper, a novel and simple defogging approach is proposed based on the estimation of depth map by considering the density of fog in local image regions. The proposed approach uses the scene-specific depth map information to compute the dark channel and transmission. The quality of recovered image is further improved by a post-processing technique. Experimental evaluation performed on FRIDA and FRIDA2 benchmark datasets demonstrates the proposed defogging framework outperforms state-of-the-art approaches. The code and the results of this work are open-sourced for reproducibility (https://github.com/RPRO5/Defogging).