Unsupervised Learning Of Depth Research Articles

3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose From Monocular Video

Depth and ego-motion estimations are essential for the localization and navigation of autonomous robots and autonomous driving. Recent studies make it possible to learn the per-pixel depth and ego-motion from the unlabeled monocular video. In this paper, a novel unsupervised training framework is proposed with 3D hierarchical refinement and augmentation using explicit 3D geometry. In this framework, the depth and pose estimations are hierarchically and mutually coupled to refine the estimated pose layer by layer. The intermediate view image is proposed and synthesized by warping the pixels in an image with the estimated depth and coarse pose. Then, the residual pose transformation can be estimated from the new view image and the image of the adjacent frame to refine the coarse pose. The iterative refinement is implemented in a differentiable manner in this paper, making the whole framework optimized uniformly. Meanwhile, a new image augmentation method is proposed for the pose estimation by synthesizing a new view image, which creatively augments the pose in 3D space but gets a new augmented 2D image. The experiments on KITTI demonstrate that our depth estimation achieves state-of-the-art performance and even surpasses recent approaches that utilize other auxiliary tasks. Our visual odometry outperforms all recent unsupervised monocular learning-based methods and achieves competitive performance to the geometry-based method, ORB-SLAM2 with back-end optimization. The source codes will be released soon at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/IRMVLab/HRANet</uri> .

Unsupervised Learning of Depth Estimation and Camera Pose With Multi-Scale GANs

Unsupervised learning methods have achieved remarkable performance in monocular depth estimation and camera pose, which mostly solve the multi-task learning problem by using their inner geometry consistency as the self-supervision signal. While most existing approaches mostly adopt the generative model to obtain the depth map prediction, so in the resolution of depth map there is room for improvement. To this end, we present our unsupervised learning architecture based on adversarial learning model, which is used for unsupervised learning of high-resolution single view depth and camera pose. Specifically, we present a multi-scale deep convolutional Generative Adversarial Network (GAN) based learning system, which consists of three networks (pose estimation network PCNN, Generator-D and Discriminator-D for depth map prediction). Furthermore, in order to generate high-resolution depth map, we propose a multi-scale GAN model (MSGAN) to decompose the hard high-quality image generation problem into more manageable sub-problems through a coarse-to-fine process. Then, we modify the overall generation architecture of GAN model by changing the down-sampling and up-sampling components to improve the quality and accuracy of the depth map prediction. Finally, in order to improve the rate of convergence, we use the Least Square Error to increase the penalty for outliers. Detailed quantitative and qualitative evaluations of the proposed framework on the KITTI dataset show that the proposed method provides better results for both pose estimation and depth recovery.

Occlusion-Aware Unsupervised Learning of Depth From 4-D Light Fields.

Depth estimation is a fundamental issue in 4-D light field processing and analysis. Although recent supervised learning-based light field depth estimation methods have significantly improved the accuracy and efficiency of traditional optimization-based ones, these methods rely on the training over light field data with ground-truth depth maps which are challenging to obtain or even unavailable for real-world light field data. Besides, due to the inevitable gap (or domain difference) between real-world and synthetic data, they may suffer from serious performance degradation when generalizing the models trained with synthetic data to real-world data. By contrast, we propose an unsupervised learning-based method, which does not require ground-truth depth as supervision during training. Specifically, based on the basic knowledge of the unique geometry structure of light field data, we present an occlusion-aware strategy to improve the accuracy on occlusion areas, in which we explore the angular coherence among subsets of the light field views to estimate initial depth maps, and utilize a constrained unsupervised loss to learn their corresponding reliability for final depth prediction. Additionally, we adopt a multi-scale network with a weighted smoothness loss to handle the textureless areas. Experimental results on synthetic data show that our method can significantly shrink the performance gap between the previous unsupervised method and supervised ones, and produce depth maps with comparable accuracy to traditional methods with obviously reduced computational cost. Moreover, experiments on real-world datasets show that our method can avoid the domain shift problem presented in supervised methods, demonstrating the great potential of our method. The code will be publicly available at https://github.com/jingjin25/LFDE-OccUnNet.

Unsupervised Monocular Depth Perception: Focusing on Moving Objects

As a flexible passive 3D sensing means, unsupervised learning of depth from monocular videos is becoming an important research topic. It utilizes the photometric errors between the target view and the synthesized views from its adjacent source views as the loss instead of the difference from the ground truth. Occlusion and scene dynamics in real-world scenes still adversely affect the learning, despite significant progress made recently. In this paper, we show that deliberately manipulating photometric errors can efficiently deal with these difficulties better. We first propose an outlier masking technique that considers the occluded or dynamic pixels as statistical outliers in the photometric error map. With the outlier masking, the network learns the depth of objects that move in the opposite direction to the camera more accurately. To the best of our knowledge, such cases have not been seriously considered in the previous works, even though they pose a high risk in applications like autonomous driving. We also propose an efficient weighted multi-scale scheme to reduce the artifacts in the predicted depth maps. Extensive experiments on the KITTI dataset and additional experiments on the Cityscapes dataset have verified the proposed approach’s effectiveness on depth or ego-motion estimation. Furthermore, for the first time, we evaluate the predicted depth on the regions of dynamic objects and static background separately for both supervised and unsupervised methods. The evaluation further verifies the effectiveness of our proposed technical approach and provides some interesting observations that might inspire future research in this direction.

Unsupervised Learning of Depth from Monocular Videos Using 3D-2D Corresponding Constraints

Depth estimation can provide tremendous help for object detection, localization, path planning, etc. However, the existing methods based on deep learning have high requirements on computing power and often cannot be directly applied to autonomous moving platforms (AMP). Fifth-generation (5G) mobile and wireless communication systems have attracted the attention of researchers because it provides the network foundation for cloud computing and edge computing, which makes it possible to utilize deep learning method on AMP. This paper proposes a depth prediction method for AMP based on unsupervised learning, which can learn from video sequences and simultaneously estimate the depth structure of the scene and the ego-motion. Compared with the existing unsupervised learning methods, our method makes the spatial correspondence among pixel points consistent with the image area by smoothing the 3D corresponding vector field based on 2D image, which effectively improves the depth prediction ability of the neural network. Our experiments on the KITTI driving dataset demonstrated that our method outperformed other previous learning-based methods. The results on the Apolloscape and Cityscapes datasets show that our proposed method has a strong universality.

Monocular Visual Odometry based on joint unsupervised learning of depth and optical flow with geometric constraints

Inferring camera ego-motion from consecutive images is essential in visual odometry (VO). In this work, we present a jointly unsupervised learning system for monocular VO, consisting of single-view depth, two-view optical flow, and camera-motion estimation module. Our work mitigates the scale drift issue which can further result in a degraded performance in the long-sequence scene. We achieve this by incorporating standard epipolar geometry into the framework. Specifically, we extract correspondences over predicted optical flow and then recover ego-motion. Additionally, we obtain pseudo-ground-truth depth via triangulating 2D-2D pixel matches, which makes the depth scale is closely relevant to the pose. Experimentation on the KITTI driving dataset shows competitive performance compared to established methods.

Unsupervised Learning of Depth and Camera Pose with Feature Map Warping.

Estimating the depth of image and egomotion of agent are important for autonomous and robot in understanding the surrounding environment and avoiding collision. Most existing unsupervised methods estimate depth and camera egomotion by minimizing photometric error between adjacent frames. However, the photometric consistency sometimes does not meet the real situation, such as brightness change, moving objects and occlusion. To reduce the influence of brightness change, we propose a feature pyramid matching loss (FPML) which captures the trainable feature error between a current and the adjacent frames and therefore it is more robust than photometric error. In addition, we propose the occlusion-aware mask (OAM) network which can indicate occlusion according to change of masks to improve estimation accuracy of depth and camera pose. The experimental results verify that the proposed unsupervised approach is highly competitive against the state-of-the-art methods, both qualitatively and quantitatively. Specifically, our method reduces absolute relative error (Abs Rel) by 0.017–0.088.

Unsupervised learning of depth and ego-motion with absolutely global scale recovery from visual and inertial data sequences

ABSTRACT In this paper, we propose an unsupervised learning method for jointly estimating monocular depth and ego-motion, which is capable to recover the absolute scale of global camera trajectory. In order to solve the general problems of scale drift and scale ambiguity of monocular camera, we fuse geometric movement data from inertial measurement unit (IMU), and use Bi-directional Long Short-Term Memory (BiLSTM) to extract temporal features. Besides, we add a lightweight and efficient attention mechanism, Convolutional Block Attention Module (CBAM), to Convolutional Neural Networks (CNNs) to complete the extraction of image features. Considering the scenes with severe illumination changes, ambiguous structures, moving objects and occlusions, especially scenes with progressively-variant textures, the geometric features can provide adaptive estimation results in the case of the degeneration of visual features. Experiments on the KITTI driving dataset reveal that our scheme achieves promising results in the estimation of camera pose and depth. Moreover, the absolute scale recovery for the global camera trajectory is effective.

Unsupervised Learning of Depth and Ego-Motion from Cylindrical Panoramic Video with Applications for Virtual Reality

We introduce a convolutional neural network model for unsupervised learning of depth and ego-motion from cylindrical panoramic video. Panoramic depth estimation is an important technology for applications such as virtual reality, 3D modeling, and autonomous robotic navigation. In contrast to previous approaches for applying convolutional neural networks to panoramic imagery, we use the cylindrical panoramic projection which allows for the use of the traditional CNN layers such as convolutional filters and max pooling without modification. Our evaluation of synthetic and real data shows that unsupervised learning of depth and ego-motion on cylindrical panoramic images can produce high-quality depth maps and that an increased field-of-view improves ego-motion estimation accuracy. We create two new datasets to evaluate our approach: a synthetic dataset created using the CARLA simulator, and Headcam, a novel dataset of panoramic video collected from a helmet-mounted camera while biking in an urban setting. We also apply our network to the problem of converting monocular panoramas to stereo panoramas.

Joint Unsupervised Learning of Depth, Pose, Ground Normal Vector and Ground Segmentation by a Monocular Camera Sensor.

We propose a completely unsupervised approach to simultaneously estimate scene depth, ego-pose, ground segmentation and ground normal vector from only monocular RGB video sequences. In our approach, estimation for different scene structures can mutually benefit each other by the joint optimization. Specifically, we use the mutual information loss to pre-train the ground segmentation network and before adding the corresponding self-learning label obtained by a geometric method. By using the static nature of the ground and its normal vector, the scene depth and ego-motion can be efficiently learned by the self-supervised learning procedure. Extensive experimental results on both Cityscapes and KITTI benchmark demonstrate the significant improvement on the estimation accuracy for both scene depth and ego-pose by our approach. We also achieve an average error of about 3 for estimated ground normal vectors. By deploying our proposed geometric constraints, the IOU accuracy of unsupervised ground segmentation is increased by 35% on the Cityscapes dataset.

Unsupervised Learning of Depth, Optical Flow and Pose With Occlusion From 3D Geometry

In autonomous driving, monocular sequences contain lots of information. Monocular depth estimation, camera ego-motion estimation and optical flow estimation in consecutive frames are high-profile concerns recently. By analyzing tasks above, pixels in the middle frame are modeled into three parts: the rigid region, the non-rigid region, and the occluded region. In joint unsupervised training of depth and pose, we can segment the occluded region explicitly. The occlusion information is used in unsupervised learning of depth, pose and optical flow, as the image reconstructed by depth-pose and optical flow will be invalid in occluded regions. A less-than-mean mask is designed to further exclude the mismatched pixels interfered with by motion or illumination change in the training of depth and pose networks. This method is also used to exclude some trivial mismatched pixels in the training of the optical flow network. Maximum normalization is proposed for depth smoothness term to restrain depth degradation in textureless regions. In the occluded region, as depth and camera motion can provide more reliable motion estimation, they can be used to instruct unsupervised learning of optical flow. Our experiments in KITTI dataset demonstrate that the model based on three regions, full and explicit segmentation of the occlusion region, the rigid region, and the non-rigid region with corresponding unsupervised losses can improve performance on three tasks significantly. The source code is available at: https://github.com/guangmingw/DOPlearning.

Unsupervised Learning of Depth and Visual Odometry using Photometric Calibration

Depth and visual odometry estimation are two essential parts in SLAM systems. Compared with traditional algorithms, supervised learning methods have shown promising results in single view depth estimation and visual odometry estimation. However, they require large amounts of labeled data. Recently, some unsupervised approaches to estimate depth and odometry via minimizing photometric error draw great attention. In this paper, we present a novel approach to learn depth and odometry via unsupervised learning. Our method ameliorates the original photometric loss to enhance the robustness to illumination change in real scenarios. In addition, we propose a new structure of Pose-net and Explainability-net to achieve rotation-sensitive odometry results and more accurate explainability masks. The experimental results have demonstrated that our approach achieves better performance than existing unsupervised methods in both depth and odometry results.

Unsupervised learning of depth estimation based on attention model and global pose optimization

Depth estimation is a fundamental task for 3D scene perception. Unsupervised learning is a prevalent method for depth estimation due to its generalization ability, and it requires no extra ground truth of depth for training. The typical pipeline for unsupervised solution uses the photometric error between target view and reconstructed view from adjacent frames as supervisory signal, in which the depth and pose are both learned and used for reconstruction. In this paper, we proposed a novel framework for unsupervised learning of depth, which consists of a Details Preserved Depth Network with attention model (DPDN), and Global Pose Calculation (GPC) modules. The attention model is adopted in the depth network to preserve the details of the depth map, which enables the network to maintain the shape of objects and enhance edges of the depth map. Moreover, instead of using the learning-based pose-network with two frames, a global pose estimation is optimized using all previous frames by mathematically minimizing the reprojection error. Experimental results show that our method outperforms existing methods in terms of accuracy and visual quality.

Unsupervised Learning of Monocular Depth and Large-Ego-Motion With Multiple Loop Consistency Losses

We propose UnLearnerVO, a jointly unsupervised learning framework for monocular depth, camera motion estimation from videos. UnLearnerVO is coupled with the relationships of 3D scene geometry and can estimate the 6-DoF pose of a monocular camera in an end-to-end pattern. There are two significant features of the proposed UnLearnerVO: one is an unsupervised depth learning pipeline based on the consecutive /inconsecutive frames, and the other is robustness in a scenario with large camera motion. Specifically, we deeply excavate the pose loop consistency loss, thus optimizing the camera pose and enforcing consistency of the estimated poses across consecutive and inconsecutive frames. Furthermore, a photometric loop consistency loss is proposed, which reduces the disturbance caused by factors such as the dynamic motion object and the photo inconsistency. The experiments on KITTI datasets show that our UnLearnerVO achieves the state-of-the-art results in large camera motion scenarios and performs better than the currently popular unsupervised approaches.

Image library for unsupervised learning of depth from stereo

Image library for unsupervised learning of depth from stereo