NYU Depth V2 Dataset Research Articles

Feature fusion and context interaction for RGB-D indoor semantic segmentation

Depth information in indoor scenes is crucial to understanding the spatial and occluding relationships of objects. Depth information contains rich geometric details and is unaffected by lighting, texture, and color, significantly enhancing traditional color images. RGB images and depth images have been widely applied in various image analysis. However, effectively leveraging the complementary information from both modalities remains a challenge. To address this issue, we propose a unified and effective feature fusion and context interaction network (FCINet). It includes three modules. Firstly, we construct a multi-modal feature fusion module (MFFM), which can achieve the aggregation of two modalities along spatial and channel dimensions. Secondly, we construct a global and local information interaction context module (GLCM) to encode rich semantic information. Finally, we construct a feature alignment and fusion module (FAFM), which integrates upsampled low-level features with high-level features to mitigate spatial information loss. Experimental results indicate that the model achieved favorable outcomes on both the NYU Depth v2 dataset and the more complex SUN RGB-D dataset.

SPDepth: Enhancing Self-Supervised Indoor Monocular Depth Estimation via Self-Propagation

Due to the existence of low-textured areas in indoor scenes, some self-supervised depth estimation methods have specifically designed sparse photometric consistency losses and geometry-based losses. However, some of the loss terms cannot supervise all the pixels, which limits the performance of these methods. Some approaches introduce an additional optical flow network to provide dense correspondences supervision, but overload the loss function. In this paper, we propose to perform depth self-propagation based on feature self-similarities, where high-accuracy depths are propagated from supervised pixels to unsupervised ones. The enhanced self-supervised indoor monocular depth estimation network is called SPDepth. Since depth self-similarities are significant in a local range, a local window self-attention module is embedded at the end of the network to propagate depths in a window. The depth of a pixel is weighted using the feature correlation scores with other pixels in the same window. The effectiveness of self-propagation mechanism is demonstrated in the experiments on the NYU Depth V2 dataset. The root-mean-squared error of SPDepth is 0.585 and the δ1 accuracy is 77.6%. Zero-shot generalization studies are also conducted on the 7-Scenes dataset and provide a more comprehensive analysis about the application characteristics of SPDepth.

TAG-fusion: Two-stage attention guided multi-modal fusion network for semantic segmentation

In the current research, leveraging auxiliary modalities, such as depth information or point cloud information, to improve RGB semantic segmentation has shown significant potential. However, existing methods mainly use convolutional modules for aggregating features from auxiliary modalities, thereby lacking sufficient exploitation of long-range dependencies. Moreover, fusion strategies are typically limited to singular approaches. In this paper, we propose a transformer-based multimodal fusion framework to better utilize auxiliary modalities for enhancing semantic segmentation results. Specifically, we employ a dual-stream architecture for extracting features from RGB and auxiliary modalities, respectively. We incorporate both early fusion and deep feature fusion techniques. At each layer, we introduce mixed attention mechanisms to leverage features from other modalities, guiding and enhancing the current modality's features before propagating them to the subsequent stage of feature extraction. After the extraction of features from different modalities, we employ an enhanced cross-attention mechanism for feature interaction, followed by channel fusion to obtain the final semantic features. Subsequently, we provide separate supervision to the network on the RGB stream, auxiliary stream, and fusion stream to facilitate the learning of representations for different modalities. The experimental results demonstrate that our framework exhibits superior performance across diverse modalities. Specifically, our approach achieves state-of-the-art results on the NYU Depth V2, SUN-RGBD, DELIVER and MFNet datasets.

LapUNet: a novel approach to monocular depth estimation using dynamic laplacian residual U-shape networks

Monocular depth estimation is an important but challenging task. Although the performance has been improved by adopting various encoder-decoder architectures, the estimated depth maps lack structure details and clear edges due to simple repeated upsampling. To solve this problem, this paper presents the novel LapUNet (Laplacian U-shape networks), in which the encoder adopts ResNeXt101, and the decoder is constructed with the novel DLRU (dynamic Laplacian residual U-shape) module. The DLRU module based on the U-shape structure can supplement high-frequency features by fusing dynamic Laplacian residual into the process of upsampling, and the residual is dynamically learnable due to the addition of convolutional operation. Also, the ASPP (atrous spatial pyramid pooling) module is introduced to capture image context at multiple scales though multiple parallel atrous convolutional layers, and the depth map fusion module is used for combining high and low frequency features from depth maps with different spatial resolution. Experiments demonstrate that the proposed model with moderate model size is superior to other previous competitors on the KITTI and NYU Depth V2 datasets. Furthermore, 3D reconstruction and target ranging by utilizing the estimated depth maps prove the effectiveness of our proposed method.

DBCAN: DFormer-Based Cross-Attention Network for RGB Depth Semantic Segmentation

Existing RGB-depth semantic segmentation methods primarily rely on symmetric two-stream Convolutional Neural Networks (CNNs) to extract RGB and spatial features separately. However, these architectures have limitations in incorporating spatial features and efficiently fusing RGB and depth information. In this study, we propose a novel architecture called the DFormer-Based Cross-Attention Network (DBCAN), which utilizes DFormer as an encoder for feature extraction and integrates several modifications to address these challenges. While DFormer is leveraged for its strong feature extraction capabilities, our modifications in the decoder focus on improving cross-modal fusion and spatial feature incorporation. We introduce three modules in the decoding process: the Object-Region Generated Module (ORGM), the Feature-Region Relation Module (FRRM), and the Spatial-Semantic Fusion Module (SSFM), which enhance feature interaction and segmentation accuracy. Experimental results on the NYUDepthv2 and SUN-RGBD datasets demonstrate that DBCAN achieves state-of-the-art performance, highlighting the effectiveness of our architectural enhancements in overcoming the limitations of existing models.

Robust Shape Fitting for 3D Scene Abstraction.

Humans perceive and construct the world as an arrangement of simple parametric models. In particular, we can often describe man-made environments using volumetric primitives such as cuboids or cylinders. Inferring these primitives is important for attaining high-level, abstract scene descriptions. Previous approaches for primitive-based abstraction estimate shape parameters directly and are only able to reproduce simple objects. In contrast, we propose a robust estimator for primitive fitting, which meaningfully abstracts complex real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to a depth map. We condition the network on previously detected parts of the scene, parsing it one-by-one. To obtain cuboids from single RGB images, we additionally optimise a depth estimation CNN end-to-end. Naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene. We thus propose an improved occlusion-aware distance metric correctly handling opaque scenes. Furthermore, we present a neural network based cuboid solver which provides more parsimonious scene abstractions while also reducing inference time. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.

A Scale-space Approach for Surface Normal Vector Estimation from Depth Maps

Surface normal vectors provide cues about the local geometric features of the scene which are utilized in many computer vision and computer graphics applications. Thus, the estimation of surface normals by utilizing structured range sensor data is an important research field. Thereupon, we propose a learning-free algorithm to estimate the surface normal vectors from depth maps. Our simple yet effective method relies on computations carried out in scale-space. Our main idea is to estimate the surface normals which cannot be properly computed in the finest scale from the coarser scales. Our method can estimate the surface normals even for images included in datasets that have challenging characteristics such as noisy real-world data or significantly large planar regions that either have a small or no gradient change. We analyze our algorithm’s performance by utilizing five benchmarks, namely, the MIT-Berkeley Intrinsic Images dataset, the New Tsukuba Dataset, the SceneNet RGB-D dataset, the IID-NORD dataset, and the NYU Depth Dataset V2, and by using two different evaluation strategies. According to the experimental results, our method can estimate surface normals efficiently without requiring neither complex computations nor huge amounts of data.

Attention-based fusion network for RGB-D semantic segmentation

RGB-D semantic segmentation can realize a profound comprehension of scenes, which is crucial in various computer vision tasks. However, due to the inherent modal variances and image noise, achieving superior segmentation using existing methods remains challenging. In this paper, we propose an attention-based fusion network for RGB-D semantic segmentation. Specifically, our network employs a forward multi-step propagation strategy and a backward progressive bootstrap fusion strategy based on the encoder–decoder architecture. By aggregating feature maps at different scales, we effectively diminish the uncertainty in the final prediction. Meanwhile, we introduce a Channel and Spatial Rectification Module (CSRM) to enable multi-dimensional interactions and noise removal. In order to achieve comprehensive integration between RGB and depth images, we put the rectified features into the Cross-Attention Fusion Module(CAFM). Extensive experiments show that our network can adeptly manage a diverse array of complex scenarios, demonstrating its innovative strength with superior performance and robust effectiveness across indoor NYU Depth V2 and SUN-RGBD datasets, and extending its capabilities to the outdoor Cityscapes dataset.

Online Scene Semantic Understanding Based on Sparsely Correlated Network for AR.

Real-world understanding serves as a medium that bridges the information world and the physical world, enabling the realization of virtual-real mapping and interaction. However, scene understanding based solely on 2D images faces problems such as a lack of geometric information and limited robustness against occlusion. The depth sensor brings new opportunities, but there are still challenges in fusing depth with geometric and semantic priors. To address these concerns, our method considers the repeatability of video stream data and the sparsity of newly generated data. We introduce a sparsely correlated network architecture (SCN) designed explicitly for online RGBD instance segmentation. Additionally, we leverage the power of object-level RGB-D SLAM systems, thereby transcending the limitations of conventional approaches that solely emphasize geometry or semantics. We establish correlation over time and leverage this correlation to develop rules and generate sparse data. We thoroughly evaluate the system's performance on the NYU Depth V2 and ScanNet V2 datasets, demonstrating that incorporating frame-to-frame correlation leads to significantly improved accuracy and consistency in instance segmentation compared to existing state-of-the-art alternatives. Moreover, using sparse data reduces data complexity while ensuring the real-time requirement of 18 fps. Furthermore, by utilizing prior knowledge of object layout understanding, we showcase a promising application of augmented reality, showcasing its potential and practicality.

SIM-MultiDepth: Self-Supervised Indoor Monocular Multi-Frame Depth Estimation Based on Texture-Aware Masking

Self-supervised monocular depth estimation methods have become the focus of research since ground truth data are not required. Current single-image-based works only leverage appearance-based features, thus achieving a limited performance. Deep learning based multiview stereo works facilitate the research on multi-frame depth estimation methods. Some multi-frame methods build cost volumes and take multiple frames as inputs at the time of test to fully utilize geometric cues between adjacent frames. Nevertheless, low-textured regions, which are dominant in indoor scenes, tend to cause unreliable depth hypotheses in the cost volume. Few self-supervised multi-frame methods have been used to conduct research on the issue of low-texture areas in indoor scenes. To handle this issue, we propose SIM-MultiDepth, a self-supervised indoor monocular multi-frame depth estimation framework. A self-supervised single-frame depth estimation network is introduced to learn the relative poses and supervise the multi-frame depth learning. A texture-aware depth consistency loss is designed considering the calculation of the patch-based photometric loss. Only the areas where multi-frame depth prediction is considered unreliable in low-texture regions are supervised by the single-frame network. This approach helps improve the depth estimation accuracy. The experimental results on the NYU Depth V2 dataset validate the effectiveness of SIM-MultiDepth. The zero-shot generalization studies on the 7-Scenes and Campus Indoor datasets aid in the analysis of the application characteristics of SIM-MultiDepth.

IIMT-net: Poly-1 weights balanced multi-task network for semantic segmentation and depth estimation using interactive information

Semantic segmentation and depth estimation are two basic researchable problems in computer vision. In common, we explore the two tasks separately. However, in some scenes, such as autonomous driving, they need be done at the same time. Meanwhile, there exists interconnected information between two tasks, which can jointly promote the performances of them. Thus, we explore the two tasks based on multi-task learning to jointly train the tasks and gain predictions together. In this paper, we build Interactive Information Multi-Task Network (IIMT-Net) incorporating the information interactive modules, trained with proposed task-balancing strategy. To be specific, we construct the principal part of encoder and decoder based on Transformer to well capture the global information. For better utilization of the task interaction between two tasks, we also add information fusion modules in two sub-decoders. In addition, the task-balancing strategy, Poly-1 weights, is designed as the balance among samples with different degrees of difficulty to ensure the network won't be biased towards any task severely. The proposed approach's exceptional performance has been extensively showcased through experimental results on the NYU Depth V2 dataset, the Cityscapes dataset, and the SUN RGB-D dataset. Our model can complete the predictions of semantic segmentation task and depth estimation task together and obtain mIoU values of 46.66% on the NYU Depth V2 dataset, 66.37% on the Cityscapes dataset, and 49.89% on the SUN RGB-D dataset, respectively with rmse values of 0.648, 6.630 and 0.401 for depth estimation task, which outperform most existing methods in multi-task learning.

Semantic Guidance Fusion Network for Cross-Modal Semantic Segmentation.

Leveraging data from various modalities to enhance multimodal segmentation tasks is a well-regarded approach. Recently, efforts have been made to incorporate an array of modalities, including depth and thermal imaging. Nevertheless, the effective amalgamation of cross-modal interactions remains a challenge, given the unique traits each modality presents. In our current research, we introduce the semantic guidance fusion network (SGFN), which is an innovative cross-modal fusion network adept at integrating a diverse set of modalities. Particularly, the SGFN features a semantic guidance module (SGM) engineered to boost bi-modal feature extraction. It encompasses a learnable semantic guidance convolution (SGC) designed to merge intensity and gradient data from disparate modalities. Comprehensive experiments carried out on the NYU Depth V2, SUN-RGBD, Cityscapes, MFNet, and ZJU datasets underscore both the superior performance and generalization ability of the SGFN compared to the current leading models. Moreover, when tested on the DELIVER dataset, the efficiency of our bi-modal SGFN displayed a mIoU that is comparable to the hitherto leading model, CMNEXT.

F[formula omitted]Depth: Self-supervised indoor monocular depth estimation via optical flow consistency and feature map synthesis

Self-supervised monocular depth estimation methods have been increasingly given much attention due to the benefit of not requiring large, labelled datasets. Such self-supervised methods require high-quality salient features and consequently suffer from severe performance drop for indoor scenes, where low-textured regions dominant in the scenes are almost indiscriminative. To address the issue, we propose a self-supervised indoor monocular depth estimation framework called F2Depth. A self-supervised optical flow estimation network is introduced to supervise depth learning. To improve optical flow estimation performance in low-textured areas, only some patches of points with more discriminative features are adopted for finetuning based on our well-designed patch-based photometric loss. The finetuned optical flow estimation network generates high-accuracy optical flow as a supervisory signal for depth estimation. Correspondingly, an optical flow consistency loss is designed. Multi-scale feature maps produced by finetuned optical flow estimation network perform warping to compute feature map synthesis loss as another supervisory signal for depth learning. Experimental results on the NYU Depth V2 dataset demonstrate the effectiveness of the framework and our proposed losses. To evaluate the generalization ability of our F2Depth, we collect a Campus Indoor depth dataset composed of approximately 1500 points selected from 99 images in 18 scenes. Zero-shot generalization experiments on 7-Scenes dataset and Campus Indoor achieve δ1 accuracy of 75.8% and 76.0% respectively. The accuracy results show that our model can generalize well to monocular images captured in unknown indoor scenes.

A Novel Method for Monocular Depth Estimation Using an Hourglass Neck Module.

In this paper, we propose a novel method for monocular depth estimation using the hourglass neck module. The proposed method has the following originality. First, feature maps are extracted from Swin Transformer V2 using a masked image modeling (MIM) pretrained model. Since Swin Transformer V2 has a different patch size for each attention stage, it is easier to extract local and global features from images input by the vision transformer (ViT)-based encoder. Second, to maintain the polymorphism and local inductive bias of the feature map extracted from Swin Transformer V2, a feature map is input into the hourglass neck module. Third, deformable attention can be used at the waist of the hourglass neck module to reduce the computation cost and highlight the locality of the feature map. Finally, the feature map traverses the neck and proceeds through a decoder, comprised of a deconvolution layer and an upsampling layer, to generate a depth image. To evaluate the objective reliability of the proposed method in this paper, we used the NYU Depth V2 dataset to compare and evaluate the methods published in other papers. As a result of the experiment, the RMSE value of the novel method for monocular depth estimation using the hourglass neck module proposed in this paper was 0.274, which was lower than those published in other papers. The lower the RMSE value, the better the depth estimation method; therefore, its efficiency compared to other techniques has been proven.

RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion.

Raw depth images captured in indoor scenarios frequently exhibit extensive missing values due to the inherent limitations of the sensors and environments. For example, transparent materials frequently elude detection by depth sensors; surfaces may introduce measurement inaccuracies due to their polished textures, extended distances, and oblique incidence angles from the sensor. The presence of incomplete depth maps imposes significant challenges for subsequent vision applications, prompting the development of numerous depth completion techniques to mitigate this problem. Numerous methods excel at reconstructing dense depth maps from sparse samples, but they often falter when faced with extensive contiguous regions of missing depth values, a prevalent and critical challenge in indoor environments. To overcome these challenges, we design a novel two-branch end-to-end fusion network named RDFC-GAN, which takes a pair of RGB and incomplete depth images as input to predict a dense and completed depth map. The first branch employs an encoder-decoder structure, by adhering to the Manhattan world assumption and utilizing normal maps from RGB-D information as guidance, to regress the local dense depth values from the raw depth map. The other branch applies an RGB-depth fusion CycleGAN, adept at translating RGB imagery into detailed, textured depth maps while ensuring high fidelity through cycle consistency. We fuse the two branches via adaptive fusion modules named W-AdaIN and train the model with the help of pseudo depth maps. Comprehensive evaluations on NYU-Depth V2 and SUN RGB-D datasets show that our method significantly enhances depth completion performance particularly in realistic indoor settings.

Monocular depth estimation using self-supervised learning with more effective geometric constraints

Self-supervised learning-based depth estimation from monocular videos is a challenging yet promising way for 3D environment perception. Existing methods that use photometric consistency as supervision are often fragile in the case of textureless environments or illumination variations. In this paper, we adopt geometric constraints to improve the reliability and accuracy of monocular depth estimation. We introduce an equation derived from the two-view imaging geometry, from which we also develop a novel geometric loss function that is shown to be more effective than the epipolar constraint in self-supervised learning. In addition, we design a depth reconstruction loss by explicitly aligning the depth scale of the consecutive estimated depth maps to keep the depth scale consistent. Finally, a new self-supervised learning framework is formulated by incorporating the proposed geometric constraints into depth, pose, and optical flow estimation models. Extensive experiments on KITTI, Make3D, and NYU Depth v2 datasets show that, by adding our geometric constraints, the depth estimation model compares favorably with state-of-the-art self-supervised learning methods, and each sub-task has a significant improvement.

Cross-Modal Transformer for RGB-D semantic segmentation of production workshop objects

Scene understanding in a production workshop is an important technology to improve its intelligence level, semantic segmentation of production workshop objects is an effective method for realizing scene understanding. Since the varieties of information of production workshop, making full use of the complementary information of RGB image and depth image can effectively improve the semantic segmentation accuracy of production workshop objects. Aiming at solving the multi-scale and real-time problems of segmenting the production workshop objects, this paper proposes Cross-Modal Transformer (CMFormer), a Transformer-based cross-modal semantic segmentation model. Its key feature correction and feature fusion parts are composed of the Multi-Scale Channel Attention Correction(MS-CAC) module and the Global Feature Aggregation(GFA) module. By improving Multi-Head Self-Attention(MHSA) in Transformer, we design Cross-Modal Multi-Head Self-Attention(CM-MHSA) to build long-range interaction between RGB image and depth image, and further design the MS-CAC module and the GFA module on the basis of the CM-MHSA module, to achieve cross-modal information interaction in the channel and spatial dimensions. Among them, the MS-CAC module enriches the multi-scale features of each channel and achieve more accurate channel attention correction between the two modals; the GFA module interacts with RGB feature and depth feature in the spatial dimension and fuses global and local features at the same time. In the experiments on the NYU Depth v2 dataset, the CMFormer reached 68.00% MPA(Mean Pixel Accuracy) and 55.75% mIoU(Mean Intersection over Union), achieves the state-of-the-art results. While in the experiments on the Scene Objects for Production workshop dataset(SOP), the CMFormer achieves 96.74% MPA, 92.98% mIoU and 43 FPS(Frames Per Second), which has high precision and good real-time performance. Code is available at: https://github.com/FutureIAI/CMFormer

Non-local affinity adaptive acceleration propagation network for generating dense depth maps from LiDAR.

Depth completion aims to generate dense depth maps from the sparse depth images generated by LiDAR. In this paper, we propose a non-local affinity adaptive accelerated (NL-3A) propagation network for depth completion to solve the mixing depth problem of different objects on the depth boundary. In the network, we design the NL-3A prediction layer to predict the initial dense depth maps and their reliability, non-local neighbors and affinities of each pixel, and learnable normalization factors. Compared with the traditional fixed-neighbor affinity refinement scheme, the non-local neighbors predicted by the network can overcome the propagation error problem of mixed depth objects. Subsequently, we combine the learnable normalized propagation of non-local neighbor affinity with pixel depth reliability in the NL-3A propagation layer, so that it can adaptively adjust the propagation weight of each neighbor during the propagation process, which enhances the robustness of the network. Finally, we design an accelerated propagation model. This model enables parallel propagation of all neighbor affinities and improves the efficiency of refining dense depth maps. Experiments on KITTI depth completion and NYU Depth V2 datasets show that our network is superior to most algorithms in terms of accuracy and efficiency of depth completion. In particular, we predict and reconstruct more smoothly and consistently at the pixel edges of different objects.

Cross-modal attention fusion network for RGB-D semantic segmentation

RGB-D semantic segmentation is crucial for robots to understand scenes. Most existing methods take depth information as an additional input, leading to cross-modal semantic segmentation networks that cannot achieve the purpose of multi-scale global and local cross-modal feature complementation. In this paper, we propose a cross-modal attention fusion network for RGB-D semantic segmentation. Specifically, we adopt a coordinate attention feature interaction module (CA-FIM) to aggregate RGB and depth features at the spatial and channel levels through the coordinate attention mechanism. Then, the gated cross-attention feature fusion module (GC-FFM) fuses the expanded modal features to achieve cross-modal global inference by the gated cross-attention mechanism. Utilizing the above two modules in four stages of the network, our framework can learn multi-modal and multi-level information to reduce the uncertainty of the final prediction. Extensive experiments on the NYU Depth V2, SUN RGB-D, and Cityscapes datasets demonstrate that our cross-modal attention fusion network is effective in RGB-D semantic segmentation for various complicated scenes.

Monocular Depth Estimation Using Res-UNet with an Attention Model

Depth maps are single image metrics that carry the information of a scene in three-dimensional axes. Accurate depth maps can recreate the 3D structure of a scene, which helps in understanding the full geometry of the objects within the scene. Depth maps can be generated from a single image or multiple images. Single-image depth mapping is also known as monocular depth mapping. Depth maps are ill-posed problems that are complex and require extensive calibration. Therefore, recent methods use deep learning to develop depth maps. We propose a new method in monocular depth estimation to develop a high-quality depth map. Our approach is based on a convolutional neural network in which we used Res-UNet with a spatial attention model to develop depth maps. The addition of an attention mechanism increases the capability of feature extraction and enhances the boundaries features. It does not add any extra parameters to the network. With our proposed model, we demonstrate that a simple CNN model aided with an attention mechanism can create high-quality depth maps with a small iteration and training time. Our model performs very well compared to the existing state-of-the-art methods on the benchmark NYU-depth v2 dataset. Our model is flexible and can be applied to any depth mapping or multi-segmentation tasks.