Monocular Depth Research Articles

Monocular depth estimation for 2D mapping of simulated environments

Monocular depth estimation for 2D mapping of simulated environments

MDEConvFormer: estimating monocular depth as soft regression based on convolutional transformer

MDEConvFormer: estimating monocular depth as soft regression based on convolutional transformer

Self-supervised learning monocular depth estimation from internet photos

Monocular depth estimation (MDE) is a fundamental problem in computer vision. Recently, self-supervised learning (SSL) approaches have attracted significant attention due to the ability to train an MDE network without ground-truth depth data. However, the performance of most existing SSL-MDE methods is yet limited by the available real training dataset, which are either binocular stereo pairs or monocular video sequences. In this paper, we propose a simple but effective generalization of SSL framework such that collections of multiple view Internet photos, a virtually unlimited source of real data, are enabled to train an MDE network. Combining the depth consistency and the mask that alleviates the interference such as moving objects, the network benefits from the real correspondences in adjacent views, thus achieving the improvement. Experiments show that the generalization of Monodepth2 via the proposed method not only leads to superior performance than itself and some data-driven MDE methods, but also stably boosts the performance of multiple state-of-the-art SSL-MDE methods. Besides, experiments on SeasonDepth, a dataset with various environmental conditions, show the good generalization capability of our proposed method.

DPDFormer: A Coarse-to-Fine Model for Monocular Depth Estimation

Monocular depth estimation attracts great attention from computer vision researchers for its convenience in acquiring environment depth information. Recently classification-based MDE methods show its promising performance and begin to act as an essential role in many multi-view applications such as reconstruction and 3D object detection. However, existed classification-based MDE models usually apply fixed depth range discretization strategy across a whole scene. This fixed depth range discretization leads to the imbalance of discretization scale among different depth ranges, resulting in the inexact depth range localization. In this article, to alleviate the imbalanced depth range discretization problem in classification-based monocular depth estimation (MDE) method we follow the coarse-to-fine principle and propose a novel depth range discretization method called depth post-discretization (DPD). Based on a coarse depth anchor roughly indicating the depth range, the DPD generates the depth range discretization adaptively for every position. The depth range discretization with DPD is more fine-grained around the actual depth, which is beneficial for locating the depth range more precisely for each scene position. Besides, to better manage the prediction of the coarse depth anchor and depth probability distribution for calculating the final depth, we design a dual-decoder transformer-based network, i.e., DPDFormer, which is more compatible with our proposed DPD method. We evaluate DPDFormer on popular depth datasets NYU Depth V2 and KITTI. The experimental results prove the superior performance of our proposed method.

Unsupervised monocular depth learning using self-teaching and contrast-enhanced SSIM loss

Unsupervised monocular depth learning using self-teaching and contrast-enhanced SSIM loss

Dual-attention-based semantic-aware self-supervised monocular depth estimation

Dual-attention-based semantic-aware self-supervised monocular depth estimation

Metasurface- and PCSEL-Based Structured Light for Monocular Depth Perception and Facial Recognition.

The novel depth-sensing system presented here revolutionizes structured light (SL) technology by employing metasurfaces and photonic crystal surface-emitting lasers (PCSELs) for efficient facial recognition in monocular depth-sensing. Unlike conventional dot projectors relying on diffractive optical elements (DOEs) and collimators, our system projects approximately 45,700 infrared dots from a compact 297-μm-dimention metasurface, drastically more spots (1.43 times) and smaller (233 times) than the DOE-based dot projector in an iPhone. With a measured field-of-view (FOV) of 158° and a 0.611° dot sampling angle, the system is lens-free and lightweight and boasts lower power consumption than vertical-cavity surface-emitting laser (VCSEL) arrays, resulting in a 5-10 times reduction in power. Utilizing a GaAs-based metasurface and a simplified optical architecture, this innovation not only addresses the drawbacks of traditional SL depth-sensing but also opens avenues for compact integration into wearable devices, offering remarkable advantages in size, power efficiency, and potential for widespread adoption.

RESIDUAL NEURAL NETWORKS FOR MONOCULAR DEPTH ESTIMATION IN NATURAL ENVIRONMENTS

ABSTRACT: Presently, depth estimation research has focused on deep learning, mainly on monocular depth estimation, because this technique has proven to be an excellent alternative for methods that use expensive sensors or require high computational consumption, providing higher performance and accuracy. Despite there being a large number of works, most have focused on indoor and urban datasets. Due to this, we review the variants of the residual networks being trained with images of natural environments with a high presence of vegetation. We proposed a new dataset specialized in natural environments using the AirSim simulation to evaluate the variants of the ResNet and ResNeXt families and verify that these can be used to detect leaves and thin branches with their corresponding depth. We conclude that a deeper residual neural network is not always the best option for monocular depth estimation. It is necessary to consider the type of network architecture, computing resources available, and the content and size of the data set. Keywords: Deep Learning, Residual Networks, Monocular Depth Estimation

A Survey of Monocular Depth Estimation based on Deep Learning

Depth information is very important for machines to perceive the environment and estimate their own state. Significant advances in robotics engineering and self-driving cars in recent decades have increased the demand for accurate depth measurements. Traditional depth estimation methods include motion structure and stereo vision matching, but these are based on the feature correspondence of multiple viewpoints, and at the same time, the predicted depth map is sparse. Depth estimation is a traditional task in computer vision that can be properly predicted by applying a variety of procedures, whereas inferring depth information from a single image is an ill-posed problem. The main objective of this paper is to provide a brief overview of the development of monocular depth estimation techniques based on deep learning. This article attempts to give an overview of supervised, unsupervised, and datasets and evaluation metrics. We conclude with a brief analysis of future developments.

IterDepth: Iterative Residual Refinement for Outdoor Self-Supervised Multi-Frame Monocular Depth Estimation

Self-supervised monocular depth estimation has been a challenging task in computer vision for a long time, and it relies on only monocular or stereo video for its supervision. To address the challenge, we propose a novel multi-frame monocular depth estimation method called IterDepth, which is based on an iterative residual refinement network. IterDepth extracts depth features from consecutive frames and computes a 3D cost volume measuring the difference between current and previous features transformed by PoseCNN (pose estimation convolutional neural network). We reformulate depth prediction as a residual learning problem, revamping the dominating depth regression to enable high-accuracy multi-frame monocular depth estimation. Specifically, we design a gated recurrent depth fusion unit that seamlessly blends depth features from the cost volume, image features, and the depth prediction. The unit updates the hidden states and refines the depth map through iterative refinement, achieving more accurate predictions than existing methods. Our experiments on the KITTI dataset demonstrate that IterDepth is 7× faster in terms of FPS (frames per second) than the recent state-of-the-art DepthFormer model with competitive performance. We also test IterDepth on the Cityscapes dataset to showcase its generalization capability in other real-world environments. Moreover, IterDepth can balance accuracy and computational efficiency by adjusting the number of refinement iterations and performs competitively with other CNN-based monocular depth estimation approaches. Code will be made publicly available.

SN 2 eRF: A Framework for Neural Radiance Fields given Sparse and Noisy Poses.

Neural Radiance Fields (NeRFs) have shown impressive capabilities in synthesizing photorealistic novel views. However, their application to room-size scenes is limited by the requirement of several hundred views with accurate poses for training. To address this challenge, we propose SN 2 eRF, a framework which can reconstruct the neural radiance field with significantly fewer views and noisy poses by exploiting multiple priors. Our key insight is to leverage both multi-view and monocular priors to constrain the optimization of NeRF in the setting of sparse and noisy pose inputs. Specifically, we extract and match key points to constrain pose optimization and use Ray Transformer with a monocular depth estimator to provide dense depth prior for geometry optimization. Benefiting from these priors, our approach achieves state-of-the-art accuracy in novel view synthesis for indoor room scenarios.

Attention-Based Grasp Detection With Monocular Depth Estimation

Attention-Based Grasp Detection With Monocular Depth Estimation

Image Intrinsic-Based Unsupervised Monocular Depth Estimation in Endoscopy.

Unsupervised monocular depth estimation plays a vital role for endoscopy-based minimally invasive surgery (MIS). However, it remains challenging due to the distinctive imaging characteristics of endoscopy which disrupt the assumption of photometric consistency, a foundation relied upon by conventional methods. Distinct from recent approaches taking image pre-processing strategy, this paper introduces a pioneering solution through intrinsic image decomposition (IID) theory. Specifically, we propose a novel end-to-end intrinsic-based unsupervised monocular depth learning framework that is comprised of an image intrinsic decomposition module and a synthesis reconstruction module. This framework seamlessly integrates IID with unsupervised monocular depth estimation, and dedicated losses are meticulously designed to offer robust supervision for network training based on this novel integration. Noteworthy, we rely on the favorable property of the resulting albedo map of IID to circumvent the challenging images characteristics instead of pre-processing the input frames. The proposed method is extensively validated on SCARED and Hamlyn datasets, and better results are obtained than state-of-the-art techniques. Beside, its generalization ability and the effectiveness of the proposed components are also validated. This innovative method has the potential to elevate the quality of 3D reconstruction in monocular endoscopy, thereby enhancing the accuracy and robustness of augmented reality navigation technology in MIS. Our code will be available at: https://github.com/bobo909/IID-SfmLearner.

Attention Mono-depth: attention-enhanced transformer for monocular depth estimation of volatile kiln burden surface

Attention Mono-depth: attention-enhanced transformer for monocular depth estimation of volatile kiln burden surface

Learn to Adapt for Self-Supervised Monocular Depth Estimation.

Monocular depth estimation is one of the fundamental tasks in environmental perception and has achieved tremendous progress by virtue of deep learning. However, the performance of trained models tends to degrade or deteriorate when employed on other new datasets due to the gap between different datasets. Though some methods utilize domain adaptation technologies to jointly train different domains and narrow the gap between them, the trained models cannot generalize to new domains that are not involved in training. To boost the transferability of self-supervised monocular depth estimation models and mitigate the issue of meta-overfitting, we train the model in the pipeline of meta-learning and propose an adversarial depth estimation task. We adopt model-agnostic meta-learning (MAML) to obtain universal initial parameters for further adaptation and train the network in an adversarial manner to extract domain-invariant representations for easing meta-overfitting. In addition, we propose a constraint to impose upon cross-task depth consistency to compel the depth estimation to be identical in different adversarial tasks, which improves the performance of our method and smoothens the training process. Experiments on four new datasets demonstrate that our method adapts quite fast to new domains. Our method trained after 0.5 epoch achieves comparable results with the state-of-the-art methods trained at least 20 epochs.

EventHDR: From Event to High-Speed HDR Videos and Beyond.

Event cameras are innovative neuromorphic sensors that asynchronously capture the scene dynamics. Due to the event-triggering mechanism, such cameras record event streams with much shorter response latency and higher intensity sensitivity compared to conventional cameras. On the basis of these features, previous works have attempted to reconstruct high dynamic range (HDR) videos from events, but have either suffered from unrealistic artifacts or failed to provide sufficiently high frame rates. In this paper, we present a recurrent convolutional neural network that reconstruct high-speed HDR videos from event sequences, with a key frame guidance to prevent potential error accumulation caused by the sparse event data. Additionally, to address the problem of severely limited real dataset, we develop a new optical system to collect a real-world dataset with paired high-speed HDR videos and event streams, facilitating future research in this field. Our dataset provides the first real paired dataset for event-to-HDR reconstruction, avoiding potential inaccuracies from simulation strategies. Experimental results demonstrate that our method can generate high-quality, high-speed HDR videos. We further explore the potential of our work in cross-camera reconstruction and downstream computer vision tasks, including object detection, panoramic segmentation, optical flow estimation, and monocular depth estimation under HDR scenarios.

Spatial-Aware Dynamic Lightweight Self-Supervised Monocular Depth Estimation

Spatial-Aware Dynamic Lightweight Self-Supervised Monocular Depth Estimation

Self-supervised Adversarial Training of Monocular Depth Estimation against Physical-World Attacks.

Monocular Depth Estimation (MDE) plays a vital role in applications such as autonomous driving. However, various attacks target MDE models, with physical attacks posing significant threats to system security. Traditional adversarial training methods, which require ground-truth labels, are not directly applicable to MDE models that lack ground-truth depth. Some self-supervised model hardening techniques (e.g., contrastive learning) overlook the domain knowledge of MDE, resulting in suboptimal performance. In this work, we introduce a novel self-supervised adversarial training approach for MDE models, leveraging view synthesis without the need for ground-truth depth. We enhance adversarial robustness against real-world attacks by incorporating L0-norm-bounded perturbation during training. We evaluate our method against supervised learning-based and contrastive learning-based approaches specifically designed for MDE. Our experiments with two representative MDE networks demonstrate improved robustness against various adversarial attacks, with minimal impact on benign performance. Our code: https://github.com/Bob-cheng/DepthModelHardening.

Physical Adversarial Attack on Monocular Depth Estimation via Shape-Varying Patches

Physical Adversarial Attack on Monocular Depth Estimation via Shape-Varying Patches

Self-Supervised Monocular Depth Estimation With Self-Perceptual Anomaly Handling.

It is attractive to extract plausible 3-D information from a single 2-D image, and self-supervised learning has shown impressive potential in this field. However, when only monocular videos are available as training data, moving objects at similar speeds to the camera can disturb the reprojection process during training. Existing methods filter out some moving pixels by comparing pixelwise photometric error, but the illumination inconsistency between frames leads to incomplete filtering. In addition, existing methods calculate photometric error within local windows, which leads to the fact that even if an anomalous pixel is masked out, it can still implicitly disturb the reprojection process, as long as it is in the local neighborhood of a nonanomalous pixel. Moreover, the ill-posed nature of monocular depth estimation makes the same scene correspond to multiple plausible depth maps, which damages the robustness of the model. In order to alleviate the above problems, we propose: 1) a self-reprojection mask to further filter out moving objects while avoiding illumination inconsistency; 2) a self-statistical mask method to prevent the filtered anomalous pixels from implicitly disturbing the reprojection; and 3) a self-distillation augmentation consistency loss to reduce the impact of ill-posed nature of monocular depth estimation. Our method shows superior performance on the KITTI dataset, especially when evaluating only the depth of potential moving objects.