Predicted Depth Maps Research Articles

LUIE: Learnable physical model-guided underwater image enhancement with bi-directional unsupervised domain adaptation

Recently, learning-based underwater enhancement (UIE) methods have made considerable progress, significantly benefiting downstream tasks such as underwater semantic segmentation and underwater depth estimation. Most existing unsupervised UIE methods utilize the atmospheric image formation model to decompose underwater images into background color, transmission map, and scene radiance. However, they rely on simplified physical models for estimating the transmission map, over-simplifying its complex formation, which results in imprecise modeling of underwater scattering effects. Additionally, supervised UIE methods heavily depend on synthetic data or ground truth, leading to limited generalization capabilities due to the substantial domain gap presented in different underwater scenarios. To tackle these challenges, we propose a Learnable physical model-guided unsupervised domain adaptation framework for Underwater Image Enhancement, dubbed LUIE. LUIE learns to predict background light, depth, and scene radiance from an underwater image. We incorporate a learnable network to estimate the transmission map based on the predicted depth map. To minimize the inter-domain gap between synthetic and real underwater images, we introduce a bi-directional domain adaptation method that alternates the background light from each domain. Experimental results demonstrate the effectiveness of our proposed method compared to existing approaches, and high-level experiment results validate that our enhanced underwater results. Experiments in real-world settings on underwater ROVs platform with NVIDIA Jetson AGX Xavier further confirm the effectiveness and efficiency of our work.

MLBSNet: Mutual Learning and Boosting Segmentation Network for RGB-D Salient Object Detection

RGB-D saliency object detection (SOD) primarily segments the most salient objects from a given scene by fusing RGB images and depth maps. Due to the inherent noise in the original depth map, fusion failures may occur, leading to performance bottlenecks. To address this issue, this paper proposes a mutual learning and boosting segmentation network (MLBSNet) for RGB-D saliency object detection, which consists of a deep optimization module (DOM), a semantic alignment module (SAM), a cross-modal integration (CMI) module, and a separate reconstruct decoder (SRD). Specifically, the deep optimization module aims to obtain optimal depth information by learning the similarity between the original and predicted depth maps. To eliminate the uncertainty of single-modal neighboring features and capture the complementary features of multiple modalities, a semantic alignment module and a cross-modal integration module are introduced. Finally, a separate reconstruct decoder based on a multi-source feature integration mechanism is constructed to overcome the accuracy loss caused by segmentation. Through comparative experiments, our method outperforms 13 existing methods on five RGB-D datasets and achieves excellent performance on four evaluation metrics.

Bridging local and global representations for self-supervised monocular depth estimation

Monocular depth estimation is a challenging problem, especially in a self-supervised manner without relying on the depth ground truth. The self-supervised approach places higher demands on the global and local feature extraction capabilities of the network. Based on this view, we propose to perform efficient hybrid feature extraction by exploiting the capacity of the Transformer and convolutional neural network to model long-range dependencies and local correlations at the same time. A bowknot-type fuser is designed to align features extracted from different sources as well as bridge global and local semantic representations. To obtain higher-quality pseudo-labels from the extracted features, we suggest the pseudo-label smoothing technique to fully utilize the multi-scale features, consequently boosting the effect of self-distillation loss as an auxiliary supervision to train the neural network. In addition, we propose pixel adaptive smoothness loss to refine the predicted depth map by introducing the image’s textural and spatial information. The suggested method is trained on the KITTI benchmark using stereo image pairs and achieves competitive depth estimation performance in contrast with previous approaches. The code and models are available at https://github.com/MaylingLin/BLGR-Depth.

Virtual reality modeling application based on multi perspective and deep learning in the new media presentation and brand building of Dongguan City memory

AbstractTo help the new media presentation and brand building of Dongguan City memory, a virtual reality modeling model based on multi‐perspective and deep learning is proposed. First, in order to address the issue of imbalanced input and output information in depth map prediction tasks, as well as poor accuracy of predicted depth map boundaries, a depth map prediction model based on multiple perspectives and deep learning is built. Then, a single perspective modeling framework is proposed to address the scarcity of perspectives in practical situations, and a dynamic fusion model is built for single‐view virtual reality scenes based on multi‐view generation networks. The results indicated that the mean square error, average relative error, and average logarithmic error were minimized at 0.52, 0.138, and 0.068, respectively. The multi‐threshold accuracy index demonstrated peak values at 0.772, 0.8821, and 0.947, respectively. The grid simplification algorithm exhibited the shortest running times at 1.57, 2.52, 3.91, and 6.53 s, respectively. Moreover, the single‐view modeling frame displayed the smallest angle distance at 0.1449, while the overlap degree of the point cloud scene reached the highest levels at 77.47, 79.49, 83.5, and 84.47, respectively. To sum up, the model has a good application effect in virtual reality modeling and positively affects virtual reality technology development.

Depth acquisition from dual-frequency fringes based on end-to-end learning

The end-to-end networks have been successfully applied in fringe projection profilometry in recent years for their high flexibility and fast speed. Most of them can predict the depth map from a single fringe. But the depth map inherits the fringe fluctuation and loses the local details of the measured object. To address this issue, an end-to-end network based on double spatially frequency fringes (dual-frequency based depth acquisition network) is proposed. To release the periodic error of the predicted depth map, a dual-branch structure is designed to learn the global contour and local details of the measured object from dual-frequency patterns. To fully exploit the contextual information of the fringe patterns, five novel modules are proposed to accomplish feature extraction, down-sampling/up-sampling, and information feeding. Ablation experiments verify the effectiveness of the presented modules. Competitive experiments demonstrate that the proposed lightweight network presents higher accuracy compared to the existing end-to-end learning algorithms. Noise immunity test and physical validation demonstrate the generalization of the network.

A Survey of Monocular Depth Estimation based on Deep Learning

Depth information is very important for machines to perceive the environment and estimate their own state. Significant advances in robotics engineering and self-driving cars in recent decades have increased the demand for accurate depth measurements. Traditional depth estimation methods include motion structure and stereo vision matching, but these are based on the feature correspondence of multiple viewpoints, and at the same time, the predicted depth map is sparse. Depth estimation is a traditional task in computer vision that can be properly predicted by applying a variety of procedures, whereas inferring depth information from a single image is an ill-posed problem. The main objective of this paper is to provide a brief overview of the development of monocular depth estimation techniques based on deep learning. This article attempts to give an overview of supervised, unsupervised, and datasets and evaluation metrics. We conclude with a brief analysis of future developments.

Scale-preserving shape reconstruction from monocular endoscope image sequences by supervised depth learning.

Reconstructing 3D shapes from images are becoming popular, but such methods usually estimate relative depth maps with ambiguous scales. A method for reconstructing a scale-preserving 3D shape from monocular endoscope image sequences through training an absolute depth prediction network is proposed. First, a dataset of synchronized sequences of RGB images and depth maps is created using an endoscope simulator. Then, a supervised depth prediction network is trained that estimates a depth map from a RGB image minimizing the loss compared to the ground-truth depth map. The predicted depth map sequence is aligned to reconstruct a 3D shape. Finally, the proposed method is applied to a real endoscope imagesequence.

Lightweight and Energy-Aware Monocular Depth Estimation Models for IoT Embedded Devices: Challenges and Performances in Terrestrial and Underwater Scenarios.

The knowledge of environmental depth is essential in multiple robotics and computer vision tasks for both terrestrial and underwater scenarios. Moreover, the hardware on which this technology runs, generally IoT and embedded devices, are limited in terms of power consumption, and therefore, models with a low-energy footprint are required to be designed. Recent works aim at enabling depth perception using single RGB images on deep architectures, such as convolutional neural networks and vision transformers, which are generally unsuitable for real-time inferences on low-power embedded hardware. Moreover, such architectures are trained to estimate depth maps mainly on terrestrial scenarios due to the scarcity of underwater depth data. Purposely, we present two lightweight architectures based on optimized MobileNetV3 encoders and a specifically designed decoder to achieve fast inferences and accurate estimations over embedded devices, a feasibility study to predict depth maps over underwater scenarios, and an energy assessment to understand which is the effective energy consumption during the inference. Precisely, we propose the MobileNetV3S75 configuration to infer on the 32-bit ARM CPU and the MobileNetV3LMin for the 8-bit Edge TPU hardware. In underwater settings, the proposed design achieves comparable estimations with fast inference performances compared to state-of-the-art methods. Moreover, we statistically proved that the architecture of the models has an impact on the energy footprint in terms of Watts required by the device during the inference. Then, the proposed architectures would be considered to be a promising approach for real-time monocular depth estimation by offering the best trade-off between inference performances, estimation error and energy consumption, with the aim of improving the environment perception for underwater drones, lightweight robots and Internet of things.

Improved deep depth estimation for environments with sparse visual cues

Most deep learning-based depth estimation models that learn scene structure self-supervised from monocular video base their estimation on visual cues such as vanishing points. In the established depth estimation benchmarks depicting, for example, street navigation or indoor offices, these cues can be found consistently, which enables neural networks to predict depth maps from single images. In this work, we are addressing the challenge of depth estimation from a real-world bird’s-eye perspective in an industry environment which contains, conditioned by its special geometry, a minimal amount of visual cues and, hence, requires incorporation of the temporal domain for structure from motion estimation. To enable the system to incorporate structure from motion from pixel translation when facing context-sparse, i.e., visual cue sparse, scenery, we propose a novel architecture built upon the structure from motion learner, which uses temporal pairs of jointly unrotated and stacked images for depth prediction. In order to increase the overall performance and to avoid blurred depth edges that lie in between the edges of the two input images, we integrate a geometric consistency loss into our pipeline. We assess the model’s ability to learn structure from motion by introducing a novel industry dataset whose perspective, orthogonal to the floor, contains only minimal visual cues. Through the evaluation with ground truth depth, we show that our proposed method outperforms the state of the art in difficult context-sparse environments.

Monocular Accumulated Depth Estimation with Recursive Feature Fusion

Existing depth estimation methods usually use an encoder-decoder structure, but different areas in the image vary in the difficulty and feature demand for depth estimation. This paper proposes a recursive feature fusion monocular depth accumulation estimation method. In the encoder, recursive feature fusion fuses multi-scale features by recursively using gated recurrent units. It extracts features that adapt to the needs of different image areas, replacing cross-layer connections. In the decoder, the depth accumulation estimation decomposes the depth reconstruction process into multiple layers. Different layers predict depth maps of specific granularity and accumulate depth estimation results. Compared with other methods, such as DPT, BTS, and DORN, the proposed method achieved competitive results on two benchmark datasets, KITTI and NYU Depth V2, with Abs Rel reaching 0.058 and 0.107, and RMSE reaching 2.411 and 0.386, respectively.

ETS-3D: An Efficient Two-Stage Framework for Stereo 3D Object Detection

We propose an efficient two-stage framework for stereo 3D object detection, called ETS-3D. Contrary to many recent approaches that rely on depth maps predicted using time-consuming stereo matching models, our approach utilizes the well-designed features to generate high-quality 3D proposals in stage-1, without explicitly exploiting predicted depth map. Specifically, we leverage pixel-wise correlation to produce normalized cost volumes to weight the left image features, and fuse multi-scale weighted features to obtain the weighted and fused features for 3D proposal generation. To maintain fast computation, only the filtered positive 3D proposals are fed into the stage-2 sub-network for further proposal refinement and quality prediction. Furthermore, we reconstruct the 3D proposal features in stage-2 to make use of different feature representations, achieving more accurate detection results. The experimental results on the KITTI 3D object detection benchmark demonstrate that our method achieves state-of-the-art performance, and can run at more than 10 fps.

CORNet: Context-Based Ordinal Regression Network for Monocular Depth Estimation

Monocular depth estimation, as one of the fundamental tasks of computer vision, plays a crucial role in three-dimensional (3D) scene understanding and perception. Usually, deep learning methods recover monocular depth maps using continuous regression manners by minimizing the errors between the ground-truth depth and the predicted depth. However, fine depth features may not be fully captured through layer-by-layer coding, which is prone to low spatial resolution depth maps and insufficient details. Furthermore, it usually converges slowly and suffers from unsatisfactory results. To tackle these issues, we propose a novel model, named context-based ordinal regression network (CORNet), to reconstruct monocular depth maps in the ordinal regression manner with context information in this paper. Firstly, we put forward a novel context-based encoder with a feature transformation (FT) module to learn context information and details from inputs, and output multi-scale feature maps. Then, we design a boundary enhancement module (BEM) with a spatial attention mechanism following each operation of feature fusion, which captures boundary features in the scene to enhance the border depth. Finally, a feature optimization module (FOM) is designed to fuse and optimize the multi-scale features and boundary features to strengthen depth learning. What’s more, we introduce an ordinal weighted inference to predict depth maps from probabilities and discretization values. Experiments and results on two challenging datasets, KITTI and NYU Depth V2, demonstrate that our proposed CORNet can estimate monocular depth maps effectively and obtain superior performance in capturing geometric features over existing methods.

Self-Supervised Monocular Depth Estimation With Multiscale Perception

Extracting 3D information from a single optical image is very attractive. Recently emerging self-supervised methods can learn depth representations without using ground truth depth maps as training data by transforming the depth prediction task into an image synthesis task. However, existing methods rely on a differentiable bilinear sampler for image synthesis, which results in each pixel in a synthetic image being derived from only four pixels in the source image and causes each pixel in the depth map to perceive only a few pixels in the source image. In addition, when calculating the photometric error between a synthetic image and its corresponding target image, existing methods only consider the photometric error within a small neighborhood of each single pixel and therefore ignore correlations between larger areas, which causes the model to tend to fall into the local optima for small patches. In order to extend the perceptual area of the depth map over the source image, we propose a novel multi-scale method that downsamples the predicted depth map and performs image synthesis at different resolutions, which enables each pixel in the depth map to perceive more pixels in the source image and improves the performance of the model. As for the locality of photometric error, we propose a structural similarity (SSIM) pyramid loss to allow the model to sense the difference between images in multiple areas of different sizes. Experimental results show that our method achieves superior performance on both outdoor and indoor benchmarks.

Unsupervised Monocular Depth Perception: Focusing on Moving Objects

As a flexible passive 3D sensing means, unsupervised learning of depth from monocular videos is becoming an important research topic. It utilizes the photometric errors between the target view and the synthesized views from its adjacent source views as the loss instead of the difference from the ground truth. Occlusion and scene dynamics in real-world scenes still adversely affect the learning, despite significant progress made recently. In this paper, we show that deliberately manipulating photometric errors can efficiently deal with these difficulties better. We first propose an outlier masking technique that considers the occluded or dynamic pixels as statistical outliers in the photometric error map. With the outlier masking, the network learns the depth of objects that move in the opposite direction to the camera more accurately. To the best of our knowledge, such cases have not been seriously considered in the previous works, even though they pose a high risk in applications like autonomous driving. We also propose an efficient weighted multi-scale scheme to reduce the artifacts in the predicted depth maps. Extensive experiments on the KITTI dataset and additional experiments on the Cityscapes dataset have verified the proposed approach’s effectiveness on depth or ego-motion estimation. Furthermore, for the first time, we evaluate the predicted depth on the regions of dynamic objects and static background separately for both supervised and unsupervised methods. The evaluation further verifies the effectiveness of our proposed technical approach and provides some interesting observations that might inspire future research in this direction.

Mobile-Based 3D Modeling: An In-Depth Evaluation for the Application in Indoor Scenarios.

Indoor environment modeling has become a relevant topic in several application fields, including augmented, virtual, and extended reality. With the digital transformation, many industries have investigated two possibilities: generating detailed models of indoor environments, allowing viewers to navigate through them; and mapping surfaces so as to insert virtual elements into real scenes. The scope of the paper is twofold. We first review the existing state-of-the-art (SoA) of learning-based methods for 3D scene reconstruction based on structure from motion (SFM) that predict depth maps and camera poses from video streams. We then present an extensive evaluation using a recent SoA network, with particular attention on the capability of generalizing on new unseen data of indoor environments. The evaluation was conducted by using the absolute relative (AbsRel) measure of the depth map prediction as the baseline metric.

DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network

Depth Completion can produce a dense depth map from a sparse input and provide a more complete 3D description of the environment. Despite great progress made in depth completion, the sparsity of the input and low density of the ground truth still make this problem challenging. In this work, we propose DenseLiDAR, a novel real-time pseudo-depth guided depth completion neural network. We exploit dense pseudo-depth map obtained from simple morphological operations to guide the network in three aspects: (1) Constructing a residual structure for the output; (2) Rectifying the sparse input data; (3) Providing dense structural loss for training the network. Thanks to these novel designs, higher performance of the output could be achieved. In addition, two new metrics for better evaluating the quality of the predicted depth map are also presented. Extensive experiments on KITTI depth completion benchmark suggest that our model is able to achieve the state-of-the-art performance at the highest frame rate of 50Hz. The predicted dense depth is further evaluated by several downstream robotic perception or positioning tasks. For the task of 3D object detection, 3~5 percent performance gains on small objects categories are achieved on KITTI 3D object detection dataset. For RGB-D SLAM, higher accuracy on vehicle's trajectory is also obtained in KITTI Odometry dataset. These promising results not only verify the high quality of our depth prediction, but also demonstrate the potential of improving the related downstream tasks by using depth completion results.

Learning Topology From Synthetic Data for Unsupervised Depth Completion

We present a method for inferring dense depth maps from images and sparse depth measurements by leveraging synthetic data to learn the association of sparse point clouds with dense natural shapes, and using the image as evidence to validate the predicted depth map. Our learned prior for natural shapes uses only sparse depth as input, not images, so the method is not affected by the covariate shift when attempting to transfer learned models from synthetic data to real ones. This allows us to use abundant synthetic data with ground truth to learn the most difficult component of the reconstruction process, which is topology estimation, and use the image to refine the prediction based on photometric evidence. Our approach uses fewer parameters than previous methods, yet, achieves the state of the art on both indoor and outdoor benchmark datasets.

Monocular Depth Estimation Based on Multi-Scale Depth Map Fusion

Monocular depth estimation is a basic task in machine vision. In recent years, the performance of monocular depth estimation has been greatly improved. However, most depth estimation networks are based on a very deep network to extract features that lead to a large amount of information lost. The loss of object information is particularly serious in the encoding and decoding process. This information loss leads to the estimated depth maps lacking object structure detail and have non-clear edges. Especially in a complex indoor environment, which is our research focus in this paper, the consequences of this loss of information are particularly serious. To solve this problem, we propose a Dense feature fusion network that uses a feature pyramid to aggregate various scale features. Furthermore, to improve the fusion effectiveness of decoded object contour information and depth information, we propose an adaptive depth fusion module, which allows the fusion network to fuse various scale depth maps adaptively to increase object information in the predicted depth map. Unlike other work predicting depth maps relying on U-NET architecture, our depth map predicted by fusing multi-scale depth maps. These depth maps have their own characteristics. By fusing them, we can estimate depth maps that not only include accurate depth information but also have rich object contour and structure detail. Experiments indicate that the proposed model can predict depth maps with more object information than other prework, and our model also shows competitive accuracy. Furthermore, compared with other contemporary techniques, our method gets state-of-the-art in edge accuracy on the NYU Depth V2 dataset.

Monocular Depth Estimation With Multi-Scale Feature Fusion

Depth estimation from a single image is a crucial but challenging task for reconstructing 3D structures and inferring scene geometry. However, most existing methods fail to extract more detailed information and estimate the distant small-scale objects well. In this paper, we propose a monocular depth estimation based on multi-scale feature fusion. Specifically, to obtain input features of different scales, we first feed the input images of different scales to pre-trained residual networks with sharing weights. Then, an attention mechanism is used to learn the salient features at different scales, which can integrate detailed information at large scale feature maps and scene information at small scale feature maps. Furthermore, inspired by the dense atrous spatial pyramid pooling in semantic segmentation, we build a multi-scale feature fusion dense pyramid to further improve the ability of the feature extraction. Last, a scale-invariant error loss is used to predict depth maps in log space. We evaluate our method on several public benchmark datasets (including NYU Depth V2 and KITTI). The experiment results show that the proposed method obtains better performance than the existing methods and achieves state-of-the-art results.

Monocular depth estimation based on deep learning: An overview

Depth information is important for autonomous systems to perceive environments and estimate their own state. Traditional depth estimation methods, like structure from motion and stereo vision matching, are built on feature correspondences of multiple viewpoints. Meanwhile, the predicted depth maps are sparse. Inferring depth information from a single image (monocular depth estimation) is an ill-posed problem. With the rapid development of deep neural networks, monocular depth estimation based on deep learning has been widely studied recently and achieved promising performance in accuracy. Meanwhile, dense depth maps are estimated from single images by deep neural networks in an end-to-end manner. In order to improve the accuracy of depth estimation, different kinds of network frameworks, loss functions and training strategies are proposed subsequently. Therefore, we survey the current monocular depth estimation methods based on deep learning in this review. Initially, we conclude several widely used datasets and evaluation indicators in deep learning-based depth estimation. Furthermore, we review some representative existing methods according to different training manners: supervised, unsupervised and semi-supervised. Finally, we discuss the challenges and provide some ideas for future researches in monocular depth estimation.