High-quality Depth Maps Research Articles

SPDET: Edge-Aware Self-Supervised Panoramic Depth Estimation Transformer With Spherical Geometry.

Panoramic depth estimation has become a hot topic in 3D reconstruction techniques with its omnidirectional spatial field of view. However, panoramic RGB-D datasets are difficult to obtain due to the lack of panoramic RGB-D cameras, thus limiting the practicality of supervised panoramic depth estimation. Self-supervised learning based on RGB stereo image pairs has the potential to overcome this limitation due to its low dependence on datasets. In this work, we propose the SPDET, an edge-aware self-supervised panoramic depth estimation network that combines the transformer with a spherical geometry feature. Specifically, we first introduce the panoramic geometry feature to construct our panoramic transformer and reconstruct high-quality depth maps. Furthermore, we introduce the pre-filtered depth-image-based rendering method to synthesize the novel view image for self-supervision. Meanwhile, we design an edge-aware loss function to improve the self-supervised depth estimation for panorama images. Finally, we demonstrate the effectiveness of our SPDET with a series of comparison and ablation experiments while achieving the state-of-the-art self-supervised monocular panoramic depth estimation. Our code and models are available at https://github.com/zcq15/SPDET.

TOF-assisted binocular vision accuracy improvement method for underwater fish size inspection.

To address problems such as the lack of accuracy in acquiring depth maps for dynamic fish 3D measurements by usual binocular vision or a time-of-flight (TOF) depth camera, a TOF-assisted binocular vision depth acquisition algorithm is used to obtain high-quality depth maps. The TOF depth energy function is designed to guide the binocular stereo matching process, which improves the correct matching rate of binocular matching in low-texture regions; the TOF and binocular stereo matching confidence weighting functions are designed to achieve the fusion of the two at pixel level to improve the matching quality of fish in the occluded overlapping regions. The experimental results show that the TOF-assisted binocular vision system improves the accuracy of fish size measurement compared to single binocular vision while reducing the measurement error when the fish body has a significant inclination along the depth axis.

A new fractional-order variational approach for joint de-hazing and segmentation of images

Segmenting outdoor images in the presence of haze, fog or smog (which fades the colors and diminishes the contrast of the observed objects) has been a challenging task in image processing with several important applications. In this paper, we propose a new fractional-order variational model that will be able to de-haze and segment a given image simultaneously. The proposed method incorporates the atmospheric veil estimation based on the dark channel prior (DCP). This transmission map can reduce significantly the edge artifacts and enhance estimation precision in the resulting image. The transmission map is then changed over to the high-quality depth map, with which the new fractional-order variational model can be framed to look for the haze free segmenting image for both grey and color outdoor images. An explicit gradient descent scheme is employed to find efficiently the minimizer of the proposed energy functional. Experimental tests on real world scenes show that the proposed method can jointly de-haze and segment hazy or foggy images effectively and efficiently.

Depth cue enhancement and guidance network for RGB-D salient object detection

Depth maps have been proven profitable to provide supplements for salient object detection in recent years. However, most RGB-D salient object detection approaches ignore that there are usually low-quality depth maps, which will inevitably result in unsatisfactory results. In this paper, we propose a depth cue enhancement and guidance network (DEGNet) for RGB-D salient object detection by exploring the depth quality enhancement and utilizing the depth cue guidance to generate predictions with highlighted objects and suppressed backgrounds. Specifically, a depth cue enhancement module is designed to generate high-quality depth maps by enhancing the contrast between the foreground and the background. Then considering the different characteristics of unimodal RGB and depth features, we use different feature enhancement strategies to strengthen the representation capability of side-output unimodal features. Moreover, we propose a depth-guided feature fusion module to excavate depth cues provided by the depth stream to guide the fusion of multi-modal features by fully making use of different modal properties, thus generating discriminative cross-modal features. Besides, we aggregate cross-modal features at different levels to obtain the final prediction by adopting a pyramid feature shrinking structure. Experimental results on six benchmark datasets demonstrate that the proposed network DEGNet outperforms 17 state-of-the-art methods.

DDL-MVS: Depth Discontinuity Learning for Multi-View Stereo Networks

We propose an enhancement module called depth discontinuity learning (DDL) for learning-based multi-view stereo (MVS) methods. Traditional methods are known for their accuracy but struggle with completeness. While recent learning-based methods have improved completeness at the cost of accuracy, our DDL approach aims to improve accuracy while retaining completeness in the reconstruction process. To achieve this, we introduce the joint estimation of depth and boundary maps, where the boundary maps are explicitly utilized for further refinement of the depth maps. We validate our idea by integrating it into an existing learning-based MVS pipeline where the reconstruction depends on high-quality depth map estimation. Extensive experiments on various datasets, namely DTU, ETH3D, “Tanks and Temples”, and BlendedMVS, show that our method improves reconstruction quality compared to our baseline, Patchmatchnet. Our ablation study demonstrates that incorporating the proposed DDL significantly reduces the depth map error, for instance, by more than 30% on the DTU dataset, and leads to improved depth map quality in both smooth and boundary regions. Additionally, our qualitative analysis has shown that the reconstructed point cloud exhibits enhanced quality without any significant compromise on completeness. Finally, the experiments reveal that our proposed model and strategies exhibit strong generalization capabilities across the various datasets.

Monocular Depth Estimation Using Res-UNet with an Attention Model

Depth maps are single image metrics that carry the information of a scene in three-dimensional axes. Accurate depth maps can recreate the 3D structure of a scene, which helps in understanding the full geometry of the objects within the scene. Depth maps can be generated from a single image or multiple images. Single-image depth mapping is also known as monocular depth mapping. Depth maps are ill-posed problems that are complex and require extensive calibration. Therefore, recent methods use deep learning to develop depth maps. We propose a new method in monocular depth estimation to develop a high-quality depth map. Our approach is based on a convolutional neural network in which we used Res-UNet with a spatial attention model to develop depth maps. The addition of an attention mechanism increases the capability of feature extraction and enhances the boundaries features. It does not add any extra parameters to the network. With our proposed model, we demonstrate that a simple CNN model aided with an attention mechanism can create high-quality depth maps with a small iteration and training time. Our model performs very well compared to the existing state-of-the-art methods on the benchmark NYU-depth v2 dataset. Our model is flexible and can be applied to any depth mapping or multi-segmentation tasks.

RGB-Guided Depth Map Recovery by Two-Stage Coarse-to-fine Dense CRF Models.

Depth maps generally suffer from large erroneous areas even in public RGB-Depth datasets. Existing learning-based depth recovery methods are limited by insufficient high-quality datasets and optimization-based methods generally depend on local contexts not to effectively correct large erroneous areas. This paper develops an RGB-guided depth map recovery method based on the fully connected conditional random field (dense CRF) model to jointly utilize local and global contexts of depth maps and RGB images. A high-quality depth map is inferred by maximizing its probability conditioned upon a low-quality depth map and a reference RGB image based on the dense CRF model. The optimization function is composed of redesigned unary and pairwise components, which constraint local structure and global structure of depth map, respectively, with the guidance of RGB image. In addition, the texture-copy artifacts problem is handled by two-stage dense CRF models in a coarse-to-fine way. A coarse depth map is first recovered by embedding RGB image in a dense CRF model in unit of 3×3 blocks. It is refined afterward by embedding RGB image in another model in unit of individual pixels and restricting the model mainly work in discontinued regions. Extensive experiments on six datasets verify that the proposed method considerably outperforms a dozen of baseline methods in correcting erroneous areas and diminishing texture-copy artifacts of depth maps.

Promoting Monocular Depth Estimation by Multi-Scale Residual Laplacian Pyramid Fusion

Deep learning approach has achieved great success in monocular depth estimation. However, the learned deep network may produce a depth map with fewer details and incorrect global depth layout, especially when the learned network is applied to a high-resolution image. In order to generate a high-quality depth map with better global structure and richer details, we propose a multi-scale residual Laplacian pyramid fusion net (MS-RLap-FNet), to fuse the multi-scale depth maps estimated by the existing depth estimation models, for depth refinement. Our approach relies on a proposed multi-scale residual Laplacian pyramid decomposition of the multi-scale depth maps, and the fusion network modules to gradually refine the depth maps based on the decomposition from low to high resolution. Comprehensive experiments show that our method, by refining the depth maps based on three popular monocular depth estimation models (DPT, MiDas, SGR), outperforms the existing state-of-the-art methods both in quantity and quality on three public datasets with different image resolutions. The depth map refined by our method has better global depth layout with richer fine details.

Monocular Depth Estimation of Old Photos via Collaboration of Monocular and Stereo Networks

Old photos that were captured about a century ago have archaeological and historical significance. Many of the old photos have been successfully digitized, but most of them suffer from severe and complicated distortion. Thus, prior studies have focused on image restoration tasks such as denoising, inpainting, and colorization. In this paper, we pay attention to the depth estimation of old photos, enabling a more enjoyable appreciation of them and helping better understand past human life, activities, and environments. Because most old photos are available as single-view images, monocular depth estimation techniques can be considered a solution. However, most high-performance techniques are based on supervised learning, which requires ground-truth depth maps. Because this kind of supervised learning is not feasible for old photos, in this paper, we present a learning framework that finetunes a pretrained monocular depth estimation network for each old photo. Specifically, the pretrained monocular depth estimation network predicts stereo depth maps for stereo image rendering. Then, the pretrained stereo network predicts depth estimates from the rendered stereo image pair. By extracting reliable depth estimates and using them for supervision of the monocular network, the monocular network can be gradually learned to produce a high-quality depth map of the given old photo. From the qualitative and quantitative performance evaluations on old photos, we demonstrate the effectiveness of the proposed method.

A Comparison of Deep Neural Networks for Monocular Depth Map Estimation in Natural Environments Flying at Low Altitude

Currently, the use of Unmanned Aerial Vehicles (UAVs) in natural and complex environments has been increasing, because they are appropriate and affordable solutions to support different tasks such as rescue, forestry, and agriculture by collecting and analyzing high-resolution monocular images. Autonomous navigation at low altitudes is an important area of research, as it would allow monitoring parts of the crop that are occluded by their foliage or by other plants. This task is difficult due to the large number of obstacles that might be encountered in the drone's path. The generation of high-quality depth maps is an alternative for providing real-time obstacle detection and collision avoidance for autonomous UAVs. In this paper, we present a comparative analysis of four supervised learning deep neural networks and a combination of two for monocular depth map estimation considering images captured at low altitudes in simulated natural environments. Our results show that the Boosting Monocular network is the best performing in terms of depth map accuracy because of its capability to process the same image at different scales to avoid loss of fine details.

Learning high-quality depth map from 360° multi-exposure imagery

Learning high-quality depth map from 360° multi-exposure imagery

Rich global feature guided network for monocular depth estimation

Monocular depth estimation is a classical but challenging task in the field of computer vision. In recent years, Convolutional Neural Network (CNN) based models have been developed to estimate high-quality depth map from a single image. Most recently, some Transformer based models have led to great improvements. All the researchers are looking for a better way to handle the global processing of information which is crucial for depth relation inference but of high computational complexity. In this paper, we take advantage of both the Transformer and CNN then propose a novel network architecture, called Rich Global Feature Guided Network (RGFN), with which rich global features are extracted from both encoder and decoder. The framework of the RGFN is the typical encoder-decoder for dense prediction. A hierarchical transformer is implemented as the encoder to capture multi-scale contextual information and model long-range dependencies. In the decoder, the Large Kernel Convolution Attention (LKCA) is adopted to extract global features from different scales and guide the network to recover fine depth maps from low spatial resolution feature maps progressively. What's more, we apply the depth-specific data augmentation method, Vertical CutDepth, to boost the performance. Experimental results on both the indoor and outdoor datasets demonstrate the superiority of the RGFN compared to other state-of-the-art models. Compared with the most recent method AdaBins, RGFN improves the RMSE score by 4.66% on the KITTI dataset and 4.67% on the NYU Depth v2 dataset.

Obstacle Avoidance and Environmental Adaptability Analysis of Snake-like Robot Based on Deep Learning

Aiming at the problems of high complexity and low accuracy of visual depth map feature recognition, a graph recognition algorithm based on principal component direction depth gradient histogram (pca-hodg) is designed in this study. In order to obtain high-quality depth map, it is necessary to calculate the parallax of the visual image. At the same time, in order to obtain the quantized regional shape histogram, it is necessary to carry out edge detection and gradient calculation on the depth map, then reduce the dimension of the depth map combined with the principal component, and use the sliding window detection method to reduce the dimension again to realize the feature extraction of the depth map. The results show that compared with other algorithms, the pca-hodg algorithm designed in this study improves the average classification accuracy and significantly reduces the average running time. This shows that the algorithm can reduce the running time by reducing the dimension, extract the depth map features more accurately, and has good robustness.

Joint Learning of Salient Object Detection, Depth Estimation and Contour Extraction

Benefiting from color independence, illumination invariance and location discrimination attributed by the depth map, it can provide important supplemental information for extracting salient objects in complex environments. However, high-quality depth sensors are expensive and can not be widely applied. While general depth sensors produce the noisy and sparse depth information, which brings the depth-based networks with irreversible interference. In this paper, we propose a novel multi-task and multi-modal filtered transformer (MMFT) network for RGB-D salient object detection (SOD). Specifically, we unify three complementary tasks: depth estimation, salient object detection and contour estimation. The multi-task mechanism promotes the model to learn the task-aware features from the auxiliary tasks. In this way, the depth information can be completed and purified. Moreover, we introduce a multi-modal filtered transformer (MFT) module, which equips with three modality-specific filters to generate the transformer-enhanced feature for each modality. The proposed model works in a depth-free style during the testing phase. Experiments show that it not only significantly surpasses the depth-based RGB-D SOD methods on multiple datasets, but also precisely predicts a high-quality depth map and salient contour at the same time. And, the resulted depth map can help existing RGB-D SOD methods obtain significant performance gain.

Self-supervised stereo depth estimation based on bi-directional pixel-movement learning.

Stereo depth estimation is an efficient method to perceive three-dimensional structures in real scenes. In this paper, we propose a novel self-supervised method, to the best of our knowledge, to extract depth information by learning bi-directional pixel movement with convolutional neural networks (CNNs). Given left and right views, we use CNNs to learn the task of middle-view synthesis for perceiving bi-directional pixel movement from left-right views to the middle view. The information of pixel movement will be stored in the features after CNNs are trained. Then we use several convolutional layers to extract the information of pixel movement for estimating a depth map of the given scene. Experiments show that our proposed method can significantly provide a high-quality depth map using only a color image as a supervisory signal.

Learning Modal and Spatial Features With Lightweight 3D Convolution for RGB Guided Depth Completion

RGB guided depth completion aims to recover a complete depth map from a sparse set of depth measurements and one corresponding RGB image, which is efficient for 3D applications to generate high-quality depth maps. Most prevailing approaches feed the sparse depth data and RGB image collected by consumer devices into a 2D convolutional network performed only at the spatial level. We argue that there is a correlation in registered multi-modal data between different modalities, which is ignored in 2D convolutional operations, resulting in loss of accuracy. In order to acquire the extra modal information between different modalities, we adopt 3D convolution for the depth completion task. Meanwhile, to decrease the significantly increased parameter size 3D convolutions, we propose a simple and effective method to reduce this increase while retaining their modal information. We verified the effectiveness of our proposed method for modal and spatial features learning on NYUv2 and KITTI depth completion datasets. Our lightweight 3D convolution achieved approximate accuracy as the standard 3D convolution, but with the same parameter size of 2D convolution. Our proposed modal features and lightweight 3D convolution are helpful to inform the development of depth sensors for consumer devices.

3D reconstruction with auto-selected keyframes based on depth completion correction and pose fusion

Dense 3D reconstruction is required for robots to safely navigate or perform advanced tasks. The accurate depth information of the image and its pose are the basis of 3D reconstruction. The resolution of depth maps obtained by LIDAR and RGB-D cameras is limited, and traditional pose calculation methods are not accurate enough. In addition, if each image is used for dense 3D reconstruction, the dense point clouds will increase the amount of calculation. To address these issues, we propose a 3D reconstruction system. Specifically, we propose a depth network of contour and gradient attention, which is used to complete and correct depth maps to obtain high-resolution and high-quality depth maps. Then, we propose a method of fusion of traditional algorithms and deep learning for pose estimation to obtain accurate localization results. Finally, we adopt the method of autonomous selection of keyframes to reduce the number of keyframes, the surfel-based geometric reconstruction is performed to reconstruct the dense 3D environment. On the TUM RGB-D, ICL-NIUM, and KITTI datasets, our method significantly improves the quality of the depth maps, the localization results, and the effect of 3D reconstruction. At the same time, we have also accelerated the speed of 3D reconstruction.

Patch-Wise Attention Network for Monocular Depth Estimation

In computer vision, monocular depth estimation is the problem of obtaining a high-quality depth map from a two-dimensional image. This map provides information on three-dimensional scene geometry, which is necessary for various applications in academia and industry, such as robotics and autonomous driving. Recent studies based on convolutional neural networks achieved impressive results for this task. However, most previous studies did not consider the relationships between the neighboring pixels in a local area of the scene. To overcome the drawbacks of existing methods, we propose a patch-wise attention method for focusing on each local area. After extracting patches from an input feature map, our module generates attention maps for each local patch, using two attention modules for each patch along the channel and spatial dimensions. Subsequently, the attention maps return to their initial positions and merge into one attention feature. Our method is straightforward but effective. The experimental results on two challenging datasets, KITTI and NYU Depth V2, demonstrate that the proposed method achieves significant performance. Furthermore, our method outperforms other state-of-the-art methods on the KITTI depth estimation benchmark.

On deep learning techniques to boost monocular depth estimation for autonomous navigation

Inferring the depth of images is a fundamental inverse problem within the field of Computer Vision since depth information is obtained through 2D images, which can be generated from infinite possibilities of observed real scenes. Benefiting from the progress of Convolutional Neural Networks (CNNs) to explore structural features and spatial image information, Single Image Depth Estimation (SIDE) is often highlighted in scopes of scientific and technological innovation, as this concept provides advantages related to its low implementation cost and robustness to environmental conditions. In the context of autonomous vehicles, state-of-the-art CNNs optimize the SIDE task by producing high-quality depth maps, which are essential during the autonomous navigation process in different locations. However, such networks are usually supervised by sparse and noisy depth data, from Light Detection and Ranging (LiDAR) laser scans, and are carried out at high computational cost, requiring high-performance Graphic Processing Units (GPUs). Therefore, we propose a new lightweight and fast supervised CNN architecture combined with novel feature extraction models which are designed for real-world autonomous navigation. We also introduce an efficient surface normals module, jointly with a simple geometric 2.5D loss function, to solve SIDE problems. We also innovate by incorporating multiple Deep Learning techniques, such as the use of densification algorithms and additional semantic, surface normals and depth information to train our framework. The method introduced in this work focuses on robotic applications in indoor and outdoor environments and its results are evaluated on the competitive and publicly available NYU Depth V2 and KITTI Depth datasets.

Salient object detection via reliability‐based depth compactness and depth contrast

It can be intuitively inferred that a high-quality depth map can be used to quickly detect the salient region in stereo vision, implying that depth information plays an essential role in stereoscopic visual attention. However, existing methods generally use the depth map as an auxiliary cue to improve the saliency detection performance. In this study, the authors present an algorithm to directly detect the salient object from a high-quality depth image. The proposed algorithm utilises a depth reliability indicator to assess the confidence of a depth image. Depth compactness, a novel feature that incorporates the depth reliability of the super-pixels, is computed as a primary salient feature. Moreover, in order to enhance another salient feature (i.e. depth contrast), they develop a coarse background filtering method to suppress background interference. Experimental results demonstrate that the proposed method performs favourably against the popular depth-aware saliency detection approaches at a lower computational cost.