Depth Estimation Model Research Articles

Monocular Depth Estimation Applied to Global Localization Over 2D Floor Plans Using Free Space Density

Indoor global localization is a critical aspect of autonomous robotic navigation. The increasing demand for service consumer-grade robots that require self-localization calls for research on methods that work with easy setup and low-cost sensors. In this paper, we propose a monocular camera-based localization of a motorized wheeled robot using a 2D floor plan as a reference map. The innovation of our method lies in using depth maps estimated from monocular images to compute the free space around the robot to be used as a measurement model in a particle filter strategy. The estimated free space density is compared to the free space density extracted from particles in the 2D floor plan. Due to the inherent imperfections of estimated depth maps, we also propose a new particle weighting approach to account for uncertainties in the depth estimation from the monocular camera. Experiments performed using real-world scenario sequences of images comparing the proposed method with RGB-D camera-based approaches demonstrate the effectiveness of the method, even for imperfect depth maps obtained with the monocular depth estimation model.

Adapting Segment Anything Model for self-supervised monocular depth estimation

Adapting Segment Anything Model for self-supervised monocular depth estimation

Scene-cGAN: A GAN for underwater restoration and scene depth estimation

Despite their wide scope of application, the development of underwater models for image restoration and scene depth estimation is not a straightforward task due to the limited size and quality of underwater datasets, as well as variations in water colours resulting from attenuation, absorption and scattering phenomena in the water column. To address these challenges, we present an all-in-one conditional generative adversarial network (cGAN) called Scene-cGAN. Our cGAN is a physics-based multi-domain model designed for image dewatering, restoration and depth estimation. It comprises three generators and one discriminator. To train our Scene-cGAN, we use a multi-term loss function based on uni-directional cycle-consistency and a novel dataset. This dataset is constructed from RGB-D in-air images using spectral data and concentrations of water constituents obtained from real-world water quality surveys. This approach allows us to produce imagery consistent with the radiance and veiling light corresponding to representative water types. Additionally, we compare Scene-cGAN with current state-of-the-art methods using various datasets. Results demonstrate its competitiveness in terms of colour restoration and its effectiveness in estimating the depth information for complex underwater scenes.

Adaptive Depth Estimation via SoftHebbLayer in a Hybrid ResNet-UNet Architecture

Abstract. Depth estimation is a key task in computer vision, critical for applications such as autonomous navigation and augmented reality. This paper introduces a novel hybrid neural network that combines ResNet and UNet architectures with a SoftHebbLayer, inspired by Hebbian learning principles, to improve depth estimation from RGB images. The ResNet backbone extracts robust hierarchical features, while the UNet decoder reconstructs fine-grained depth maps. The SoftHebbLayer dynamically adjusts feature connections based on co-activation, enhancing the models adaptability to diverse environments. This approach addresses common challenges in depth estimation, including poor generalization and computational inefficiency. We evaluated the model on the DIODE dataset, achieving strong results in Mean Squared Error (MSE), which is 0.0800 and Root Mean Squared Error (RMSE), which is 0.2805, demonstrating improved accuracy in both indoor and outdoor scenes. While the model excels in precision, further refinement is needed to reduce computational overhead and improve performance in challenging environments. This research paves the way for more efficient, adaptable depth estimation models, with potential applications in mobile robotics and real-time edge computing systems.

Learn to Adapt for Self-Supervised Monocular Depth Estimation.

Monocular depth estimation is one of the fundamental tasks in environmental perception and has achieved tremendous progress by virtue of deep learning. However, the performance of trained models tends to degrade or deteriorate when employed on other new datasets due to the gap between different datasets. Though some methods utilize domain adaptation technologies to jointly train different domains and narrow the gap between them, the trained models cannot generalize to new domains that are not involved in training. To boost the transferability of self-supervised monocular depth estimation models and mitigate the issue of meta-overfitting, we train the model in the pipeline of meta-learning and propose an adversarial depth estimation task. We adopt model-agnostic meta-learning (MAML) to obtain universal initial parameters for further adaptation and train the network in an adversarial manner to extract domain-invariant representations for easing meta-overfitting. In addition, we propose a constraint to impose upon cross-task depth consistency to compel the depth estimation to be identical in different adversarial tasks, which improves the performance of our method and smoothens the training process. Experiments on four new datasets demonstrate that our method adapts quite fast to new domains. Our method trained after 0.5 epoch achieves comparable results with the state-of-the-art methods trained at least 20 epochs.

Self-supervised monocular depth estimation based on improved dense network and wavelet decomposition

Traditional self-supervised monocular depth estimation models do not adequately extract and fuse shallow features, which can easily lead to problems such as missed detection of small objects and blurred object edges. To address the above problems, this paper proposes a self-supervised monocular depth estimation model based on improved dense network and wavelet decomposition. The model follows the structure of U-net, in which the encoder adopts an improved dense connection method to improve the feature extraction and fusion capabilities of the encoder; at the same time, a detail enhancement module is added to the jump connection to further refine and integrate the multi-scale features output by the encoder; finally, the wavelet decomposition method is introduced in the decoder, forcing the decoder to pay more attention to high-frequency information and achieve refined processing of image edges. Experimental results show that the depth estimation model proposed in this paper has a stronger ability to capture small object features, and the edges of the generated depth map are clearer and more accurate.

BEW-YOLOv8: A deep learning model for multi-scene and multi-scale flood depth estimation

As global climate change intensifies, the frequency of extreme weather events, including urban flooding, has risen, posing a significant threat to the safety of urban areas. Efficient and precise flood depth monitoring is crucial for a dynamic understanding of disaster conditions, enabling informed decision-making and reducing the potential for significant loss of life and property. Although object detection models have shown promise in estimating flood depths and have achieved notable results, enhancements are still needed in applications across multi-scene and multi-scale. This research enhances the YOLOv8 model by integrating Bidirectional Feature Pyramid Network (BiFPN), Effective Squeeze and Excitation (EffectiveSE), and Wise-IoU (WIoU), targeting submerged cars to develop the BEW-YOLOv8 model. Testing shows that the BEW-YOLOv8 model improves Precision, Recall, F1 Score, mAP@0.5, and mAP@0.5–0.95 by 14.1%, 2.2%, 7.9%, 8.7%, and 6.3%, respectively, compared to the unmodified YOLOv8 model. Compared with other existing models, BEW-YOLOv8 achieves the best performance with less than one-tenth of the parameters. Furthermore, this study verifies that the BEW-YOLOv8 model holds considerable promise for further enhancements with dataset expansions. Its precision and dependability make the BEW-YOLOv8 model well-suited for real-time flood depth assessment across multiple scenes and scales. The model’s capability for real-time processing supports urgent flood response needs, offering solid technical assistance for managing urban floods and mitigating disaster risks.

Canopy coverage of wheat measured with high accuracy using the HSV colour model and relative depth estimation model, MiDaS

ABSTRACT Canopy coverage calculated from 2D images effectively estimates several parameters, such as dry weight above the ground, stem number, and risk of lodging, without destroying the crops. This study aimed to clarify how to accurately calculate canopy coverage, regardless of environmental conditions, using various colour models and a relative depth estimation model, MiDaS. For wheat cultivated in 2022/23, the canopy coverage at the initiation of the stem elongation stage (CC30), calculated using only the HSV colour model, was overestimated because of the presence of weeds. However, when the heatmap generated using MiDaS was combined with the results extracted from the HSV colour model, it showed a CC30 similar to the accurate CC30 calculated by hand drawing, regardless of the wheat variety, sowing pattern, and location of the image taken, with an RMSE as low as 3.48. The CC30 calculated by combining the HSV colour model and MiDaS was significantly positively correlated with the number of stems, dry weight above ground, and nitrogen content above ground at the initiation of the stem elongation stage of ‘Sanukinoyume 2009’ grown in 2021/22 and 2022/23 (n = 72, r > 0.798). The combination of the HSV colour model and MiDaS effectively and accurately calculates canopy coverage regardless of environmental conditions and can be used for growth diagnosis and variable nitrogen fertiliser application after the initiation of the stem elongation stage.

Learning Domain Invariant Features for Unsupervised Indoor Depth Estimation Adaptation

Predicting depth maps from monocular images has made an impressive performance in the past years. However, most depth estimation methods are trained with paired image-depth map data or multi-view images (e.g., stereo pair and monocular sequence), which suffer from expensive annotation costs and poor transferability. Although unsupervised domain adaptation methods are introduced to mitigate the reliance on annotated data, rare works focus on the unsupervised cross-scenario indoor monocular depth estimation. In this article, we propose to study the generalization of depth estimation models across different indoor scenarios in an adversarial-based domain adaptation paradigm. Concretely, a domain discriminator is designed for discriminating the representation from source and target domains, while the feature extractor aims to confuse the domain discriminator by capturing domain-invariant features. Further, we reconstruct depth maps from latent representations with the supervision of labeled source data. As a result, the feature extractor learned features possess the merit of both domain-invariant and low source risk, and the depth estimator can deal with the domain shift between source and target domains. We conduct the cross-scenario and cross-dataset experiments on the ScanNet and NYU-Depth-v2 datasets to verify the effectiveness of our method and achieve impressive performance.

Reusable Architecture Growth for Continual Stereo Matching.

The remarkable performance of recent stereo depth estimation models benefits from the successful use of convolutional neural networks to regress dense disparity. Akin to most tasks, this needs gathering training data that covers a number of heterogeneous scenes at deployment time. However, training samples are typically acquired continuously in practical applications, making the capability to learn new scenes continually even more crucial. For this purpose, we propose to perform continual stereo matching where a model is tasked to 1) continually learn new scenes, 2) overcome forgetting previously learned scenes, and 3) continuously predict disparities at inference. We achieve this goal by introducing a Reusable Architecture Growth (RAG) framework. RAG leverages task-specific neural unit search and architecture growth to learn new scenes continually in both supervised and self-supervised manners. It can maintain high reusability during growth by reusing previous units while obtaining good performance. Additionally, we present a Scene Router module to adaptively select the scene-specific architecture path at inference. Comprehensive experiments on numerous datasets show that our framework performs impressively in various weather, road, and city circumstances and surpasses the state-of-the-art methods in more challenging cross-dataset settings. Further experiments also demonstrate the adaptability of our method to unseen scenes, which can facilitate end-to-end stereo architecture learning and practical deployment.

UDR-GS: Enhancing Underwater Dynamic Scene Reconstruction with Depth Regularization

Representing and rendering dynamic underwater scenes present significant challenges due to the medium’s inherent properties, which result in image blurring and information ambiguity. To overcome these challenges and accomplish real-time rendering of dynamic underwater environments while maintaining efficient training and storage, we propose Underwater Dynamic Scene Reconstruction Gaussian Splatting (UDR-GS), a method based on Gaussian Splatting. By leveraging prior information from a pre-trained depth estimation model and smoothness constraints between adjacent images, our approach uses the estimated depth as a geometric prior to aid in color-based optimization, significantly reducing artifacts and improving geometric accuracy. By integrating depth guidance into the Gaussian Splatting (GS) optimization process, we achieve more precise geometric estimations. To ensure higher stability, smoothness constraints are applied between adjacent images, maintaining consistent depth for neighboring 3D points in the absence of boundary conditions. The symmetry concept is inherently applied in our method by maintaining uniform depth and color information across multiple viewpoints, which enhances the reconstruction quality and visual coherence. Using 4D Gaussian Splatting (4DGS) as a baseline, our strategy demonstrates superior performance in both RGB novel view synthesis and 3D geometric reconstruction. On average, across multiple datasets, our method shows an improvement of approximately 1.41% in PSNR and a 0.75% increase in SSIM compared with the baseline 4DGS method, significantly enhancing the visual quality and geometric fidelity of dynamic underwater scenes.

Redefining Accuracy: Underwater Depth Estimation for Irregular Illumination Scenes.

Acquiring underwater depth maps is essential as they provide indispensable three-dimensional spatial information for visualizing the underwater environment. These depth maps serve various purposes, including underwater navigation, environmental monitoring, and resource exploration. While most of the current depth estimation methods can work well in ideal underwater environments with homogeneous illumination, few consider the risk caused by irregular illumination, which is common in practical underwater environments. On the one hand, underwater environments with low-light conditions can reduce image contrast. The reduction brings challenges to depth estimation models in accurately differentiating among objects. On the other hand, overexposure caused by reflection or artificial illumination can degrade the textures of underwater objects, which is crucial to geometric constraints between frames. To address the above issues, we propose an underwater self-supervised monocular depth estimation network integrating image enhancement and auxiliary depth information. We use the Monte Carlo image enhancement module (MC-IEM) to tackle the inherent uncertainty in low-light underwater images through probabilistic estimation. When pixel values are enhanced, object recognition becomes more accessible, allowing for a more precise acquisition of distance information and thus resulting in more accurate depth estimation. Next, we extract additional geometric features through transfer learning, infusing prior knowledge from a supervised large-scale model into a self-supervised depth estimation network to refine loss functions and a depth network to address the overexposure issue. We conduct experiments with two public datasets, which exhibited superior performance compared to existing approaches in underwater depth estimation.

Smart insect-computer hybrid robots empowered with enhanced obstacle avoidance capabilities using onboard monocular camera

Insect-computer hybrid robots are receiving increasing attention as a potential alternative to small artificial robots due to their superior locomotion capabilities and low manufacturing costs. Controlling insect-computer hybrid robots to travel through terrain littered with complex obstacles of various shapes and sizes is still challenging. While insects can inherently deal with certain obstacles by using their antennae to detect and avoid obstacles, this ability is limited and can be interfered with by control signals when performing navigation tasks, ultimately leading to the robot being trapped in a specific place and having difficulty escaping. Hybrid robots need to add additional sensors to provide accurate perception and early warning of the external environment to avoid obstacles before getting trapped, ensuring smooth navigation tasks in rough terrain. However, due to insects’ tiny size and limited load capacity, hybrid robots are very limited in the sensors they can carry. A monocular camera is suitable for insect-computer hybrid robots because of its small size, low power consumption, and robust information acquisition capabilities. This paper proposes a navigation algorithm with an integrated obstacle avoidance module using a monocular camera for the insect-computer hybrid robot. The monocular cameras equipped with a monocular depth estimation algorithm based on deep learning can produce depth maps of environmental obstacles. The navigation algorithm generates control commands that can drive the hybrid robot away from obstacles according to the distribution of obstacle distances in the depth map. To ensure the performance of the monocular depth estimation model when applied to insect-computer hybrid robotics scenarios, we collected the first dataset from the viewpoint of a small robot for model training. In addition, we propose a simple but effective depth map processing method to obtain obstacle avoidance commands based on the weighted sum method. The success rate of the navigation experiment is significantly improved from 6.7% to 73.3%. Experimental results show that our navigation algorithm can detect obstacles in advance and guide the hybrid robots to avoid them before they get trapped.

Multiple prior representation learning for self-supervised monocular depth estimation via hybrid transformer

Self-supervised monocular depth estimation aims to infer depth information without relying on labeled data. However, the lack of labeled information poses a significant challenge to the model’s representation, limiting its ability to capture the intricate details of the scene accurately. Prior information can potentially mitigate this issue, enhancing the model’s understanding of scene structure and texture. Nevertheless, solely relying on a single type of prior information often falls short when dealing with complex scenes, necessitating improvements in generalization performance. To address these challenges, we introduce a novel self-supervised monocular depth estimation model that leverages multiple priors to bolster representation capabilities across spatial, context, and semantic dimensions. Specifically, we employ a hybrid transformer and a lightweight pose network to obtain long-range spatial priors in the spatial dimension. Then, the context prior attention is designed to improve generalization, particularly in complex structures or untextured areas. In addition, semantic priors are introduced by leveraging semantic boundary loss, and semantic prior attention is supplemented, further refining the semantic features extracted by the decoder. Experiments on three diverse datasets demonstrate the effectiveness of the proposed model. It integrates multiple priors to comprehensively enhance the representation ability, improving the accuracy and reliability of depth estimation. Codes are available at: https://github.com/MVME-HBUT/MPRLNet.

Exploring the vulnerability of self-supervised monocular depth estimation models

Recent advancements in deep learning have substantially boosted the performance of monocular depth estimation (MDE), an essential component in fully-vision-based autonomous driving systems (e.g., Tesla and Toyota). These advancements, however, have been primarily directed towards optimizing performance, with limited consideration for security vulnerabilities. This study introduces the first backdoor attack against self-supervised MDE models. By conceptualizing backdoor attacks as a multi-task challenge, we propose a novel attack framework that employs multi-task learning to iteratively optimize the standard MDE subtask and the backdoor learning subtask. Within this framework, we design two types of backdoor attacks: Target Disappearing Backdoor Attack (TDBA) and Depth Increasing Backdoor Attack (DIBA). TDBA renders specific objects invisible to the compromised model, while DIBA dramatically inflates the depth of trigger-associated objects. Extensive evaluations on three mainstream MDE models show that TDBA can make the target model directly ignore the specific objects, with a mean depth error under 0.06 meters, while DIBA significantly increases the depth for trigger-associated objects, resulting in a mean depth error exceeding 94.2 meters. These findings highlight the urgent need for advanced security measures in the development of MDE models, to mitigate such vulnerabilities.

Depth estimation and 3D reconstruction from UAV-borne imagery: Evaluation on the UseGeo dataset

Depth estimation and 3D model reconstruction from aerial imagery is an important task in photogrammetry, remote sensing, and computer vision. To compare the performance of different image-based approaches, this study presents a benchmark for UAV-based aerial imagery using the UseGeo dataset. The contributions include the release of various evaluation routines on GitHub, as well as a comprehensive comparison of baseline approaches, such as methods for offline multi-view 3D reconstruction resulting in point clouds and triangle meshes, online multi-view depth estimation, as well as single-image depth estimation using self-supervised deep learning. With the release of our evaluation routines, we aim to provide a universal protocol for the evaluation of depth estimation and 3D reconstruction methods on the UseGeo dataset. The conducted experiments and analyses show that each method excels in a different category: the depth estimation from COLMAP outperforms that of the other approaches, ACMMP achieves the lowest error and highest completeness for point clouds, while OpenMVS produces triangle meshes with the lowest error. Among the online methods for depth estimation, the approach from the Plane-Sweep Library outperforms the FaSS-MVS approach, while the latter achieves the lowest processing time. And even though the particularly challenging nature of the dataset and the small amount of training data leads to a significantly higher error in the results of the self-supervised single-image depth estimation approach, it outperforms all other approaches in terms of processing time and frame rate. In our evaluation, we have also considered modern learning-based approaches that can be used for image-based 3D reconstruction, such as NeRFs. However, due to the significantly lower quality of the resulting 3D models, we have only included a qualitative comparison between NeRF-based and conventional approaches in the scope of this work.

DME: Unveiling the Bias for Better Generalized Monocular Depth Estimation

This paper aims to design monocular depth estimation models with better generalization abilities. To this end, we have conducted quantitative analysis and discovered two important insights. First, the Simulation Correlation phenomenon, commonly seen in long-tailed classification problems, also exists in monocular depth estimation, indicating that the imbalanced depth distribution in training data may be the cause of limited generalization ability. Second, the imbalanced and long-tail distribution of depth values extends beyond the dataset scale, and also manifests within each individual image, further exacerbating the challenge of monocular depth estimation. Motivated by the above findings, we propose the Distance-aware Multi-Expert (DME) depth estimation model. Unlike prior methods that handle different depth range indiscriminately, DME adopts a divide-and-conquer philosophy where each expert is responsible for depth estimation of regions within a specific depth range. As such, the depth distribution seen by each expert is more uniform and can be more easily predicted. A pixel-level routing module is further designed and learned to stitch the prediction of all experts into the final depth map. Experiments show that DME achieves state-of-the-art performance on both NYU-Depth v2 and KITTI, and also delivers favorable zero-shot generalization capability on unseen datasets.

PPEA-Depth: Progressive Parameter-Efficient Adaptation for Self-Supervised Monocular Depth Estimation

Self-supervised monocular depth estimation is of significant importance with applications spanning across autonomous driving and robotics. However, the reliance on self-supervision introduces a strong static-scene assumption, thereby posing challenges in achieving optimal performance in dynamic scenes, which are prevalent in most real-world situations. To address these issues, we propose PPEA-Depth, a Progressive Parameter-Efficient Adaptation approach to transfer a pre-trained image model for self-supervised depth estimation. The training comprises two sequential stages: an initial phase trained on a dataset primarily composed of static scenes, succeeded by an expansion to more intricate datasets involving dynamic scenes. To facilitate this process, we design compact encoder and decoder adapters to enable parameter-efficient tuning, allowing the network to adapt effectively. They not only uphold generalized patterns from pre-trained image models but also retain knowledge gained from the preceding phase into the subsequent one. Extensive experiments demonstrate that PPEA-Depth achieves state-of-the-art performance on KITTI, CityScapes and DDAD datasets.

Irregular boundaries stereo images dataset creating using depth estimation model

AbstractThis paper introduces a stereoscopic image and depth dataset created using a deep learning model. It addresses the challenge of obtaining accurate and annotated stereo image pairs with irregular boundaries for deep learning model training. Stereoscopic image and depth dataset provides a unique resource for training deep learning models to handle irregular boundary stereoscopic images, which are valuable for real-world scenarios with complex shapes or occlusions. The dataset is created using monocular depth estimation, a state-of-the-art depth estimation model, and it can be used in applications like rectifying images, estimating depth, detecting objects, and autonomous driving. Overall, this paper presents a novel dataset that demonstrates its effectiveness and potential for advancing stereo vision and developing deep learning models for computer vision applications.

A synthetic digital city dataset for robustness and generalisation of depth estimation models

Existing monocular depth estimation driving datasets are limited in the number of images and the diversity of driving conditions. The images of datasets are commonly in a low resolution and the depth maps are sparse. To overcome these limitations, we produce a Synthetic Digital City Dataset (SDCD) which was collected under 6 different weather driving conditions, and 6 common adverse perturbations caused by the data transmission. SDCD provides a total of 930 K high-resolution RGB images and corresponding perfect observed depth maps. The evaluation shows that depth estimation models which are trained on SDCD provide a clearer, smoother, and more precise long-range depth estimation compared to those trained on one of the best-known driving datasets KITTI. Moreover, we provide a benchmark to investigate the performance of depth estimation models in different adverse driving conditions. Instead of collecting data from the real world, we generate the SDCD under severe driving conditions with perfect observed data in the digital world, enhancing depth estimation for autonomous driving.