NYU Depth V2 Dataset Research Articles

Multi-stage information diffusion for joint depth and surface normal estimation

Depth and surface normal estimations are important for 3D geometric perception, which has numerous applications including autonomous vehicles and robots. In this paper, we propose a lightweight Multi-stage Information Diffusion Network (MIDNet) for the simultaneous prediction of depth and surface normals from a single RGB image. To obtain semantic and detail-preserving features, we adopt a high-resolution network as our backbone to learn multi-scale features, which are then fused into shared features for the two tasks. To mutually boost each task, a Cross-Correlation Attention Module (CCAM) is proposed to adaptively integrate information for the prediction of the two tasks in multiple stages, including feature-level information interaction and task-level information interaction. Ablation studies show that the proposed multi-stage information diffusion strategy can boost the performance gain for the two tasks at different levels. Compared to current state-of-the-art methods on the NYU Depth V2, Stanford 2D-3D-Semantic and KITTI datasets, our method achieves superior performance for both monocular depth and surface normal estimation.

THCANet: Two-layer hop cascaded asymptotic network for robot-driving road-scene semantic segmentation in RGB-D images

In several existing red–green–blue and depth (RGB-D) semantic segmentation algorithms, schemes are used to supplement contextual information through multilayer feature interactions. However, these approaches ignore the complementation of the contextual information and the introduction of noise interfering with the segmentation process. To minimize noise interference during this process, we introduce a two-layer hop cascaded asymptotic network (THCANet) for robot-driving road-scene semantic segmentation in RGB-D images. To exploit the depth map and supervision to strengthen semantic segmentation, we propose an attention cross-fusion module for the interactive combination of RGB-D features through multimodality weighting. Notably, the information of the two modalities reduces noise during fusion. After fusing features from the RGB-D modalities, we also use a novel multiscale context module to fuse features at multiple scales and employ a jump cascade architecture between the modules to recover lost context information and suppress irrelevant noise. Moreover, multiple supervision is performed at different segmentation stages to improve accuracy. The proposed THCANet system demonstrates the best performance on a robot-driving road dataset compared with similar methods, and its generalization ability is demonstrated using the NYU-Depth V2 dataset.

Monocular Depth Estimation of Noncooperative Spacecraft Based on Deep Learning

Estimating position and stereographic geometry information of noncooperative spacecraft through three-dimensional (3-D) reconstruction techniques is of great significance. Depth information acquisition is an important component of 3-D reconstruction. Monocular cameras are cheaper and more widely used than depth sensors. The relative positions of noncooperative spacecraft and our spacecraft can be calculated from depth maps of monocular depth estimation and the camera parameters, providing data support for subsequent tracking and capture missions. This paper proposes a monocular depth estimation network combining the convolutional neural network (CNN) and a vision transformer (VIT) to improve the prediction accuracy of few-shot samples. We extract detail features and global features from the CNN and VIT encoders, respectively, and then fuse deep features and shallow features by a skip-connected upsampling decoder. Compared with the representative depth estimation algorithms in recent years on the NYU-Depth V2 dataset, the proposed network structure combines the advantages of the CNN and VIT as well as estimates the global depth of the scene more accurately while maintaining details. To solve the lack of spacecraft data collection, a new dataset is made from 3-D simulation models. Experiments on the self-made dataset demonstrate the feasibility of this method in aerospace engineering.

Removal of Redundant Information via Discrete Representation for Monocular Depth Estimation

Monocular depth estimation aims at inferring three-dimensional (3D) cues from a single RGB image. Although existing methods have achieved a certain degree of success, the impact of redundant information has rarely been studied. We propose to improve estimation accuracy by implicitly eliminating redundant information. To this end, we creatively apply discrete representation to monocular depth estimation. By mapping continuous variables into the corresponding learning-based discrete latent space, a hierarchical multi-scale latent map is acquired as the decoder input. Removing redundant information can enhance prediction performance by making the depth estimator balance the local and global. Furthermore, to fully take advantage of the discrete representation, a lightweight fusion mechanism is introduced to aggregate information in multi-scale feature maps.Experiments on NYU Depth V2 dataset demonstrate that our network is competitive with the state of the arts.

SABV-Depth: A biologically inspired deep learning network for monocular depth estimation

Monocular depth estimation makes it possible for machines to perceive the real world. The prediction performance of the depth estimation network based on deep learning will be affected due to the depth of the deep network and the locality of convolution operations. The imitation of the biological visual system and its functional structure is becoming a research hotspot. In this paper, we study the interpretability relationship between the biological visual system and the monocular depth estimation network. By concretizing the attention mechanism in biological vision, we propose a monocular depth estimation network based on the self-attention mechanism, named SABV-Depth, which can improve prediction accuracy. Inspired by the biological visual interaction mechanism, we focus on the information transfer between each module of the network and improve the information retention ability, and enable the network to output a depth map with rich object information and detailed information. Further, a decoder module with an inner-connection is proposed to recover depth maps with sharp edge contours. Our method is experimentally validated on the KITTI dataset and NYU Depth V2 dataset. The results show that compared with other works, the proposed method improves prediction accuracy. Meanwhile, the depth map has more object information and detail information, and a better edge information processing effect.

Monocular Depth Estimation Using a Laplacian Image Pyramid with Local Planar Guidance Layers.

It is important to estimate the exact depth from 2D images, and many studies have been conducted for a long period of time to solve depth estimation problems. Recently, as research on estimating depth from monocular camera images based on deep learning is progressing, research for estimating accurate depths using various techniques is being conducted. However, depth estimation from 2D images has been a problem in predicting the boundary between objects. In this paper, we aim to predict sophisticated depths by emphasizing the precise boundaries between objects. We propose a depth estimation network with encoder-decoder structures using the Laplacian pyramid and local planar guidance method. In the process of upsampling the learned features using the encoder, the purpose of this step is to obtain a clearer depth map by guiding a more sophisticated boundary of an object using the Laplacian pyramid and local planar guidance techniques. We train and test our models with KITTI and NYU Depth V2 datasets. The proposed network constructs a DNN using only convolution and uses the ConvNext networks as a backbone. A trained model shows the performance of the absolute relative error (Abs_rel) 0.054 and root mean square error (RMSE) 2.252 based on the KITTI dataset and absolute relative error (Abs_rel) 0.102 and root mean square error 0.355 based on the NYU Depth V2 dataset. On the state-of-the-art monocular depth estimation, our network performance shows the fifth-best performance based on the KITTI Eigen split and the eighth-best performance based on the NYU Depth V2.

Real-Time Monocular Depth Estimation Merging Vision Transformers on Edge Devices for AIoT

Depth estimation is requisite to build 3D perceiving capability of artificial intelligence of things (AIoT). Real-time inference with extremely low computing resource consumption is critical on edge devices. However, most single-view depth estimation networks focus on the improvement of accuracy running on high-end GPUs, which go opposite to the real-time requirement on edge devices. To address this issue, this article proposed a novel encoder-decoder network to realize real-time monocular depth estimation on edge devices. The proposed network merges semantic information at global field via an efficient transformer-based module to provide more details of the object for depth assignment. The transformer-based module is integrated in the lowest level resolution of an encoder-decoder architecture to largely reduce the parameters of the Vision Transformer (ViT). In particular, we proposed a novel patch convolutional layer for low-latency feature extraction in the encoder and a SConv5 layer for effective depth assignment in the decoder. The proposed network achieves outstanding balance between accuracy and speed on the NYU Depth v2 dataset. A low RMSE of 0.554 and a fast speed of 58.98 FPS on NVIDIA Jetson Nano device with TensorRT optimization are obtained on NYU Depth v2, outperforming most state-of-the-art real-time results.

TP-GAN: Simple Adversarial Network With Additional Player for Dense Depth Image Estimation

We present a simple yet robust monocular depth estimation technique by synthesizing a depth map image from a single RGB input image using the advantage of generative adversarial networks (GAN). We employ an additional sub-model termed refiner to extract local depth features, then combine it with the global scene information from the generator to improve the GAN’s performance compared to the standard GAN architectural scheme. Notably, the generator is the first player to learn to synthesize depth images. The second player, the discriminator, classifies the generated depth. In the meantime, the third player, the refiner, enhances the final reconstructed depth. Complementing the GAN model, we apply a conditional generative network (cGAN) to lead the generator in mapping the input image to the respective depth representation. We further incorporate a structured similarity (SSIM) as our loss function for the generator and refiner in GAN training. Through extensive experiment validation, we confirmed the performance of our strategy on the publicly indoor NYU Depth v2 and KITTI outdoor data. Experiment results on the NYU depth v2 dataset show that our proposed approach achieves the best performance by 96.0% on threshold accuracy ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\delta &lt; 1.25^{2}$ </tex-math></inline-formula> ) and the second-best accuracy on all thresholds on the KITTI dataset. We discovered that our proposed method compares favorably to numerous existing monocular depth estimation strategies and demonstrates a considerable improvement in the accuracy of image depth estimation despite its simple network architecture.

HI-Net: Boosting Self-Supervised Indoor Depth Estimation via Pose Optimization

Pose estimation plays a critical role in self-supervised monocular depth estimation for indoor scenes, especially those involving complex ego-motion. In this letter, we leverage the two-view geometry constraints into pose estimation to boost the accuracy of pose estimation, which ultimately improves the performance of self-supervised depth estimation. Specifically, we decompose pose estimation into two steps: initial homography estimation and iterative residual refinement. We first introduce a Homography Estimation Module (HEM) to estimate large 3-DoF rotations. Then, we refine the 6-DoF residual pose estimation with an Iterative Residual Refinement Module (IRM). Finally, the supervision signal is generated with the refined pose and used for the training of DepthNet. Experiments on the NYU depth V2 dataset show that our pose estimation approach significantly improves the performance of DepthNet, and the proposed method achieves state-of-the-art depth estimation results. Furthermore, experiments on the ScanNet dataset demonstrate the generalization ability of our method for both pose estimation and depth estimation.

MobileXNet: An Efficient Convolutional Neural Network for Monocular Depth Estimation

Depth estimation from a single RGB image has attracted great interest in autonomous driving and robotics. State-of-the-art methods are usually designed on top of complex and extremely deep network architectures, which require more computational resources. Moreover, the inherent characteristic of the backbone used by the existing approaches results in severe spatial information loss in the produced feature maps, which impairs the accuracy of depth estimation on small sized images. In this study, we aimed to design a novel and efficient Convolutional Neural Network (CNN) to address these problems. Specifically, we stacked two shallow encoder-decoder style subnetworks successively in a unified network. Extensive experiments have been conducted on the NYU depth v2, KITTI, Make3D and Unreal data sets. Experimental results show that the proposed network achieves comparable accuracy to state-of-the-art methods that have extremely deep architectures but runs at a much faster speed on a single, less powerful GPU.

Exploration of Semantic Label Decomposition and Dataset Size in Semantic Indoor Scenes Synthesis via Optimized Residual Generative Adversarial Networks.

In this paper, we revisit the paired image-to-image translation using the conditional generative adversarial network, the so-called "Pix2Pix", and propose efficient optimization techniques for the architecture and the training method to maximize the architecture's performance to boost the realism of the generated images. We propose a generative adversarial network-based technique to create new artificial indoor scenes using a user-defined semantic segmentation map as an input to define the location, shape, and category of each object in the scene, exactly similar to Pix2Pix. We train different residual connections-based architectures of the generator and discriminator on the NYU depth-v2 dataset and a selected indoor subset from the ADE20K dataset, showing that the proposed models have fewer parameters, less computational complexity, and can generate better quality images than the state of the art methods following the same technique to generate realistic indoor images. We also prove that using extra specific labels and more training samples increases the quality of the generated images; however, the proposed residual connections-based models can learn better from small datasets (i.e., NYU depth-v2) and can improve the realism of the generated images in training on bigger datasets (i.e., ADE20K indoor subset) in comparison to Pix2Pix. The proposed method achieves an LPIPS value of 0.505 and an FID value of 81.067, generating better quality images than that produced by Pix2Pix and other recent paired Image-to-image translation methods and outperforming them in terms of LPIPS and FID.

An Efficient Information-Reinforced Lidar Deep Completion Network without RGB Guided

Due to the sparsity of point clouds obtained by LIDAR, the depth information is usually not complete and dense. The depth completion task is to recover dense depth information from sparse depth information. However, most of the current deep completion networks use RGB images as guidance, which are more like a processing method of information fusion. They are not valid when there is only sparse depth data and no other color information. Therefore, this paper proposes an information-reinforced completion network for a single sparse depth input. We use a multi-resolution dense progressive fusion structure to maximize the multi-scale information and optimize the global situation by point folding. At the same time, we re-aggregate the confidence and impose another depth constraint on the pixel depth to make the depth estimation closer to the ground trues. Our experimental results on KITTI and NYU Depth v2 datasets show that the proposed network achieves better results than other unguided deep completion methods. And it is excellent in both accuracy and real-time performance.

MDSNet: self-supervised monocular depth estimation for video sequences using self-attention and threshold mask

Solving the problem that features prediction of self-supervised monocular depth estimation is still ambiguous in low-texture regions and boundaries. We proposed an innovative self-supervised monocular depth estimation method, monocular depth self-supervised network, which integrates three effective strategies to construct an innovative self-supervised monocular depth estimation framework: (1) the attention mechanism and feature fusion module are adopted to enhance the semantic and spatial information of feature images, (2) the threshold segmentation mask is utilized to solve object motion and low-texture regions to increase image details, and (3) the residual pose module and deep reconstruction loss are used to enhance the feature extraction capability of the model to improve the accuracy of depth and pose estimation. Comprehensive experiments and visual analysis results demonstrate the effectiveness of each component in isolation. Compared to existing self-supervised methods, our model not only achieves outstanding results on KITTI and NYU Depth V2 datasets but also can be suitable to different environments.

SIDA-GAN: A lightweight Generative Adversarial Network for Single Image Depth Approximation

SIDA-GAN is a set of GAN based deep learning models that can be employed for predicting the depth map for a given single input RGB image. This article explores the research gap present in the state of the art which lacks GAN based depth predictors implemented using Depthwise Separable Convolution (DSC). DSC reduces the number of computations in comparison with standard convolution for the same convolution task, therefore making the model computationally efficient. The adversarial loss applied by the GAN model make the predicted depth image globally coherent while loss functions like L1, SSIM fine tunes the local feature predictions leading to enhanced prediction quality. In this proposal, the depth predictor (SIDA-GAN generator) is an encoder-decoder design featuring DSC and skip connections. SIDA-GAN has two GAN variants namely conditional GAN (cGAN), patch GAN (pGAN) and two generator implementations based on MobileNetV1, MobileNetV2 leading to four different flavours. Those are SIDA-cGAN (MobileNetV1 based generator), SIDA-cGAN (MobileNetV2 based generator), SIDA-pGAN (MobileNetV1 based generator), SIDA-pGAN (MobileNetV2 based generator). The proposed models are trained with different combinations of SSIM, L1, depth-smoothness and adversarial losses using NYU-Depth V2 (Nathan Silberman et al., 2012) dataset. SIDA-GAN has achieved reduction in trainable parameter count up to 76% and performance improvement in Average Relative Error (8%–25%), Root Mean Square Error (3%–27%), threshold accuracy (2%–6%) when compared with baseline papers.

Rich global feature guided network for monocular depth estimation

Monocular depth estimation is a classical but challenging task in the field of computer vision. In recent years, Convolutional Neural Network (CNN) based models have been developed to estimate high-quality depth map from a single image. Most recently, some Transformer based models have led to great improvements. All the researchers are looking for a better way to handle the global processing of information which is crucial for depth relation inference but of high computational complexity. In this paper, we take advantage of both the Transformer and CNN then propose a novel network architecture, called Rich Global Feature Guided Network (RGFN), with which rich global features are extracted from both encoder and decoder. The framework of the RGFN is the typical encoder-decoder for dense prediction. A hierarchical transformer is implemented as the encoder to capture multi-scale contextual information and model long-range dependencies. In the decoder, the Large Kernel Convolution Attention (LKCA) is adopted to extract global features from different scales and guide the network to recover fine depth maps from low spatial resolution feature maps progressively. What's more, we apply the depth-specific data augmentation method, Vertical CutDepth, to boost the performance. Experimental results on both the indoor and outdoor datasets demonstrate the superiority of the RGFN compared to other state-of-the-art models. Compared with the most recent method AdaBins, RGFN improves the RMSE score by 4.66% on the KITTI dataset and 4.67% on the NYU Depth v2 dataset.

Monocular Depth Estimation: Lightweight Convolutional and Matrix Capsule Feature-Fusion Network.

This paper reports a study that aims to solve the problem of the weak adaptability to angle transformation of current monocular depth estimation algorithms. These algorithms are based on convolutional neural networks (CNNs) but produce results lacking in estimation accuracy and robustness. The paper proposes a lightweight network based on convolution and capsule feature fusion (CNNapsule). First, the paper introduces a fusion block module that integrates CNN features and matrix capsule features to improve the adaptability of the network to perspective transformations. The fusion and deconvolution features are fused through skip connections to generate a depth image. In addition, the corresponding loss function is designed according to the long-tail distribution, gradient similarity, and structural similarity of the datasets. Finally, the results are compared with the methods applied to the NYU Depth V2 and KITTI datasets and show that our proposed method has better accuracy on the C1 and C2 indices and a better visual effect than traditional methods and deep learning methods without transfer learning. The number of trainable parameters required by this method is 65% lower than that required by methods presented in the literature. The generalization of this method is verified via the comparative testing of the data collected from the internet and mobile phones.

Dynamic Spatial Propagation Network for Depth Completion

Image-guided depth completion aims to generate dense depth maps with sparse depth measurements and corresponding RGB images. Currently, spatial propagation networks (SPNs) are the most popular affinity-based methods in depth completion, but they still suffer from the representation limitation of the fixed affinity and the over smoothing during iterations. Our solution is to estimate independent affinity matrices in each SPN iteration, but it is over-parameterized and heavy calculation.This paper introduces an efficient model that learns the affinity among neighboring pixels with an attention-based, dynamic approach. Specifically, the Dynamic Spatial Propagation Network (DySPN) we proposed makes use of a non-linear propagation model (NLPM). It decouples the neighborhood into parts regarding to different distances and recursively generates independent attention maps to refine these parts into adaptive affinity matrices. Furthermore, we adopt a diffusion suppression (DS) operation so that the model converges at an early stage to prevent over-smoothing of dense depth. Finally, in order to decrease the computational cost required, we also introduce three variations that reduce the amount of neighbors and attentions needed while still retaining similar accuracy. In practice, our method requires less iteration to match the performance of other SPNs and yields better results overall. DySPN outperforms other state-of-the-art (SoTA) methods on KITTI Depth Completion (DC) evaluation by the time of submission and is able to yield SoTA performance in NYU Depth v2 dataset as well.

Multi-scale depth classification network for monocular depth estimation

In addition to the RGB information of an image, depth information is the most critical. Monocular depth estimation is an effective method for predicting depth from RGB images. First, we propose a multiscale classification network that transforms the predicted depth values into a nonlinear combination problem between multiple depth interval values. Based on the correlation of the depth information, a depth classification network was used to determine the results of the weight values of each interval. Second, the depth maps predicted by feature maps with different resolutions contain different critical information. The lower-resolution depth maps play an overall role in predicting the overall contour; the higher-resolution depth map is more accurate in predicting object edge details. Finally, the desired depth map can be obtained by a 3 × 3 convolution of the predicted depth maps at multiple scales. We tested our proposed method on the NYU Depth V2 and KITTI datasets and achieved effective performance.

Iterative Feature Matching for Self-Supervised Indoor Depth Estimation

In this paper, we propose an iterative feature matching framework for self-supervised depth estimation in indoor scenes. Conventional methods usually leverage the structure-from-motion supervision to help the photometric optimization escape from the local minima, which have complex ego-motion and large regions with non-texture or repeated-texture. However, the supervision is limited as the reconstruction is usually sparse. To address this, we propose an iterative feature matching framework called IFMNet to jointly learn depths and search for correspondences. With the predicted depths from the previous iteration, we present an online optimized grid searching algorithm to find more accurate correspondences. Given these new correspondences, we compute the triangulated depths and improve the depth network with adaptive bin-wise online hard example mining. Experimental results on the NYU Depth V2 and SceneNet datasets verify the effectiveness of our approach.

Single image depth estimation based on sculpture strategy

Supervised deep learning approaches have achieved state-of-the-art performance in single image depth estimation. However, the procedure to train the network is not intervened, due to the black box character of deep learning. To enhance the generalization ability of trained models, we employ a learned prior that is inspired by the process of sculpturing, to guide the model’s learning. From the view of sculpturing, the depth estimation can be divided into two consecutive steps: the initial contour estimation and the detail refinement. The contour of scene can be introduced as the intermediate supervision which enables the network to learn true knowledge and accelerate training. In addition, the edges of the scene contain much semantic information. To better use this structured knowledge, we use an attention mechanism to integrate the edges of the scene to guide training. We also create a more precise ground truth by using self-adaptive Gaussian interpolation. Compared to other datasets, our dataset is denser, and the details of small objects and the ordinal relation in the depths of different objects is preserved. Additionally, the distribution of depth values does not change. The experimental results on the KITTI, Make3D and NYU Depth V2 datasets show that the method achieved better performance with respect to other state-of-the-art depth estimation approaches. Code is available at https://github.com/XTU-PR-LAB/HG-Net.