Ground Truth Depth Research Articles

Weakly-supervised Depth Estimation and Image Deblurring via Dual-Pixel Sensors.

Dual-pixel (DP) imaging sensors are getting more popularly adopted by modern cameras. A DP camera captures a pair of images in a single snapshot by splitting each pixel in half. Several previous studies show how to recover depth information by treating the DP pair as an approximate stereo pair. However, dual-pixel disparity occurs only in image regions with defocus blur which is unlike classic stereo disparity. Heavy defocus blur in DP pairs affects the performance of depth estimation approaches based on matching. Therefore, we treat the blur removal and the depth estimation as a joint problem. We investigate the formation of the DP pair, which links the blur and depth information, rather than blindly removing the blur effect. We propose a mathematical DP model that can improve depth estimation by the blur. This exploration motivated us to propose our previous work, an end-to-end DDDNet (DP-based Depth and Deblur Network), which jointly estimates depth and restores the image in a supervised fashion. However, collecting the ground-truth (GT) depth map for the DP pair is challenging and limits the depth estimation potential of the DP sensor. Therefore, we propose an extension of the DDDNet, called WDDNet (Weakly-supervised Depth and Deblur Network), which includes an efficient reblur solver that does not require GT depth maps for training. To achieve this, we convert all-in-focus images into supervisory signals for unsupervised depth estimation in our WDDNet. We jointly estimate an all-in-focus image and a disparity map, then use a Reblur and Fstack module to regularize the disparity estimation and image restoration. We conducted extensive experiments on synthetic and real data to demonstrate the competitive performance of our method when compared to state-of-the-art (SOTA) supervised approaches. Index Terms-Dual-pixel Sensor, Weakly-supervised, Depth Estimation, Deblur and Reblu.

Graph semantic information for self-supervised monocular depth estimation

Self-supervised monocular depth estimation has garnered significant attention in recent years due to its practical value in applications, as it eliminates the need for ground truth depth maps during training. However, its performance usually drops when estimating weakly textured regions and boundary regions, primarily due to the limited depth representation capability of traditional Convolutional Neural Networks (CNNs) that do not support topology. To address these issues, we propose a Graph Semantic Model (GSM) to improve self-supervised monocular depth estimation by utilizing graph learning and semantic information. Our focus is on improving feature representation through graph semantic information. Therefore, we incorporate semantic segmentation and depth estimation into one framework and enhance the interaction of different modal information through the Inter-Directed Graph Reasoning (IDGR) module. In addition, we design the Semantic-Guided Edge Graph Reasoning (SGEGR) module, aiming to boost the network's ability to perceive local depth. Extensive experiments on the KITTI dataset show that our method outperforms the state-of-the-art methods, particularly in accurately estimating depth within weakly textured regions and boundary regions.

GDM-depth: Leveraging global dependency modelling for self-supervised indoor depth estimation

Self-supervised depth estimation algorithms eschew depth ground truth and employ the convolutional U-Net with a fixed receptive field which confines its focus primarily to nearby spatial distances. These factors obscure adequate supervision during image reconstruction, consequently hindering accurate depth estimation, particularly in complex indoor scenes. The pure transformer framework can perform global modelling to provide more semantic information. However, the cost is significant. To tackle these challenges, we introduce GDM-Depth, which utilizes global dependency modelling to offer more precise depth guidance from the network itself. Initially, we propose integrating learnable tree filters with unary terms, leveraging the structural properties of spanning trees to facilitate efficient long-range interactions. Subsequently, instead of replacing the convolutional framework entirely, we employ the transformer to design a scale-aware global feature extractor, establishing global relationships among local features at various scales, achieving both efficiency and cost-effectiveness. Furthermore, inter-class disparities between depth global and local features are observed. To address this issue, we introduce the global feature injector to further enhance the representation. GDM-Depth's effectiveness is demonstrated on the NYUv2, ScanNet, and InteriorNet depth datasets, achieving impressive test set performances of 87.2%, 83.1%, and 76.1% in key indicators δ<0.125, respectively.

MonoLoT: Self-Supervised Monocular Depth Estimation in Low-Texture Scenes for Automatic Robotic Endoscopy.

The self-supervised monocular depth estimation framework is well-suited for medical images that lack ground-truth depth, such as those from digestive endoscopes, facilitating navigation and 3D reconstruction in the gastrointestinal tract. However, this framework faces several limitations, including poor performance in low-texture environments, limited generalisation to real-world datasets, and unclear applicability in downstream tasks like visual servoing. To tackle these challenges, we propose MonoLoT, a self-supervised monocular depth estimation framework featuring two key innovations: point matching loss and batch image shuffle. Extensive ablation studies on two publicly available datasets, namely C3VD and SimCol, have shown that methods enabled by MonoLoT achieve substantial improvements, with accuracies of 0.944 on C3VD and 0.959 on SimCol, surpassing both depth-supervised and self-supervised baselines on C3VD. Qualitative evaluations on real-world endoscopic data underscore the generalisation capabilities of our methods, outperforming both depth-supervised and self-supervised baselines. To demonstrate the feasibility of using monocular depth estimation for visual servoing, we have successfully integrated our method into a proof-of-concept robotic platform, enabling real-time automatic intervention and control in digestive endoscopy. In summary, our method represents a significant advancement in monocular depth estimation for digestive endoscopy, overcoming key challenges and opening promising avenues for medical applications.

Selfredepth

Depth maps produced by consumer-grade sensors suffer from inaccurate measurements and missing data from either system or scene-specific sources. Data-driven denoising algorithms can mitigate such problems; however, they require vast amounts of ground truth depth data. Recent research has tackled this limitation using self-supervised learning techniques, but it requires multiple RGB-D sensors. Moreover, most existing approaches focus on denoising single isolated depth maps or specific subjects of interest highlighting a need for methods that can effectively denoise depth maps in real-time dynamic environments. This paper extends state-of-the-art approaches for depth-denoising commodity depth devices, proposing SelfReDepth, a self-supervised deep learning technique for depth restoration, via denoising and hole-filling by inpainting of full-depth maps captured with RGB-D sensors. The algorithm targets depth data in video streams, utilizing multiple sequential depth frames coupled with color data to achieve high-quality depth videos with temporal coherence. Finally, SelfReDepth is designed to be compatible with various RGB-D sensors and usable in real-time scenarios as a pre-processing step before applying other depth-dependent algorithms. Our results demonstrate our approach’s real-time performance on real-world datasets shows that it outperforms state-of-the-art methods in denoising and restoration performance at over 30 fps on Commercial Depth Cameras, with potential benefits for augmented and mixed-reality applications.

Development of large-scale synthetic 3D point cloud datasets for vision-based bridge structural condition assessment

Visual recognition of 3D point cloud data of bridge inspection scenes is a key step in automating the visual inspection process, which is currently largely manual and inefficient. To alleviate the lack of large-scale annotated point cloud datasets for training such 3D visual recognition algorithms, this research investigates an approach for developing large-scale synthetic point cloud datasets. The proposed approach proceeds in four steps: (1) random generation of different types of bridges in computer graphics environments; (2) sampling of camera trajectories that represent data collection scenarios during bridge inspection; (3) 3D reconstruction using Structure from Motion (SfM) applied to rendered synthetic images; (4) automated annotation of the reconstructed point cloud using ground truth masks obtained with synthetic images. Besides, this research proposes to store point uncertainty information defined by the error between the ground truth depth and the depth calculated from the SfM results. Prior to training, thresholds can be applied to this uncertainty information to control the levels of outliers in the dataset. This research demonstrates the proposed approach by generating point cloud datasets for two data collection scenarios. The effectiveness of the generated datasets is investigated by training 3D semantic segmentation algorithms and evaluating the performance on real and synthetic point cloud data. The proposed approach for point cloud dataset generation will facilitate the development of generalizable and high level-of-detail 3D recognition algorithms toward autonomous bridge inspection.

Self-Supervised Monocular Depth Estimation via Binocular Geometric Correlation Learning

Monocular depth estimation aims to infer a depth map from a single image. Although supervised learning-based methods have achieved remarkable performance, they generally rely on a large amount of labor-intensively annotated data. Self-supervised methods, on the other hand, do not require any annotation of ground-truth depth and have recently attracted increasing attention. In this work, we propose a self-supervised monocular depth estimation network via binocular geometric correlation learning. Specifically, considering the inter-view geometric correlation, a binocular cue prediction module is presented to generate the auxiliary vision cue for the self-supervised learning of monocular depth estimation. Then, to deal with the occlusion in depth estimation, an occlusion interference attenuated constraint is developed to guide the supervision of the network by inferring the occlusion region and producing paired occlusion masks. Experimental results on two popular benchmark datasets have demonstrated that the proposed network obtains competitive results compared to state-of-the-art self-supervised methods and achieves comparable results to some popular supervised methods.

Unsupervised Monocular Depth Estimation With Channel and Spatial Attention.

Understanding 3-D scene geometry from videos is a fundamental topic in visual perception. In this article, we propose an unsupervised monocular depth and camera motion estimation framework using unlabeled monocular videos to overcome the limitation of acquiring per-pixel ground-truth depth at scale. The photometric loss couples the depth network and pose network together and is essential to the unsupervised method, which is based on warping nearby views to target using the estimated depth and pose. We introduce the channelwise attention mechanism to dig into the relationship between channels and introduce the spatialwise attention mechanism to utilize the inner-spatial relationship of features. Both of them applied in depth networks can better activate the feature information between different convolutional layers and extract more discriminative features. In addition, we apply the Sobel boundary to our edge-aware smoothness for more reasonable accuracy, and clearer boundaries and structures. All of these help to close the gap with fully supervised methods and show high-quality state-of-the-art results on the KITTI benchmark and great generalization performance on the Make3D dataset.

CSRNet: Focusing on critical points for depth completion

Depth completion is an effective method for generating dense depth maps from sparse ones. In recent studies, the majority of points, which we call standard points, often exhibit sub-optimal performance. This issue arises from the need to fit only very few points, termed as challenging points, which consist of noises and regions with discontinuous depth in the ground-truth. On the other hand, traditional evaluations can not recognize this situation, and are dominated by these limited challenging points, whose performance improvements may not significantly benefit related tasks. In contrast, standard points, which are critical for these tasks, are not effectively measured. This discrepancy highlights the need for a more targeted approach and evaluation method for depth completion. In order to solve the above problems, we propose a standard-point-enhancing learning paradigm. This paradigm aims to improve the performance on standard points, which consists of a Cascaded Segmentation-to-Regression Networks (CSRNet) and a Mining L1 loss. CSRNet includes two branches: DSNet and DRNet. DSNet uses segmentation to generate a coarse depth map, providing challenging-point-insensitive information. DRNet adopts a coarse-to-fine approach to learn the residual depth map between the coarse depth map and the ground-truth depth map. In addition, our Mining L1 loss leverages the segmentation results to filter out potential challenging points. This approach allows the network to concentrate more effectively on standard points. Lastly, we introduce the Minimum Error (ME) Curves as a new way to measure the performance of predicted depth maps in a flexible and comprehensive manner, irrespective of whether the points are standard or challenging. Experimental results on the KITTI and NYUDv2 datasets show that our approach significantly improves accuracy on the majority of points.

Bridging local and global representations for self-supervised monocular depth estimation

Monocular depth estimation is a challenging problem, especially in a self-supervised manner without relying on the depth ground truth. The self-supervised approach places higher demands on the global and local feature extraction capabilities of the network. Based on this view, we propose to perform efficient hybrid feature extraction by exploiting the capacity of the Transformer and convolutional neural network to model long-range dependencies and local correlations at the same time. A bowknot-type fuser is designed to align features extracted from different sources as well as bridge global and local semantic representations. To obtain higher-quality pseudo-labels from the extracted features, we suggest the pseudo-label smoothing technique to fully utilize the multi-scale features, consequently boosting the effect of self-distillation loss as an auxiliary supervision to train the neural network. In addition, we propose pixel adaptive smoothness loss to refine the predicted depth map by introducing the image’s textural and spatial information. The suggested method is trained on the KITTI benchmark using stereo image pairs and achieves competitive depth estimation performance in contrast with previous approaches. The code and models are available at https://github.com/MaylingLin/BLGR-Depth.

Enhancing View Synthesis with Depth-Guided Neural Radiance Fields and Improved Depth Completion.

Neural radiance fields (NeRFs) leverage a neural representation to encode scenes, obtaining photorealistic rendering of novel views. However, NeRF has notable limitations. A significant drawback is that it does not capture surface geometry and only renders the object surface colors. Furthermore, the training of NeRF is exceedingly time-consuming. We propose Depth-NeRF as a solution to these issues. Specifically, our approach employs a fast depth completion algorithm to denoise and complete the depth maps generated by RGB-D cameras. These improved depth maps guide the sampling points of NeRF to be distributed closer to the scene's surface, benefiting from dense depth information. Furthermore, we have optimized the network structure of NeRF and integrated depth information to constrain the optimization process, ensuring that the termination distribution of the ray is consistent with the scene's geometry. Compared to NeRF, our method accelerates the training speed by 18%, and the rendered images achieve a higher PSNR than those obtained by mainstream methods. Additionally, there is a significant reduction in RMSE between the rendered scene depth and the ground truth depth, which indicates that our method can better capture the geometric information of the scene. With these improvements, we can train the NeRF model more efficiently and achieve more accurate rendering results.

Self-supervised Adversarial Training of Monocular Depth Estimation against Physical-World Attacks.

Monocular Depth Estimation (MDE) plays a vital role in applications such as autonomous driving. However, various attacks target MDE models, with physical attacks posing significant threats to system security. Traditional adversarial training methods, which require ground-truth labels, are not directly applicable to MDE models that lack ground-truth depth. Some self-supervised model hardening techniques (e.g., contrastive learning) overlook the domain knowledge of MDE, resulting in suboptimal performance. In this work, we introduce a novel self-supervised adversarial training approach for MDE models, leveraging view synthesis without the need for ground-truth depth. We enhance adversarial robustness against real-world attacks by incorporating L0-norm-bounded perturbation during training. We evaluate our method against supervised learning-based and contrastive learning-based approaches specifically designed for MDE. Our experiments with two representative MDE networks demonstrate improved robustness against various adversarial attacks, with minimal impact on benign performance. Our code: https://github.com/Bob-cheng/DepthModelHardening.

MosaicMVS: Mosaic-Based Omnidirectional Multi-View Stereo for Indoor Scenes

We present MosaicMVS, a novel learning-based depth estimation framework for a mosaic-based omnidirectional multi-view stereo (MVS) camera setup. It uses a regular field of view (FOV) MVS network for an omnidirectional imaging setup with explicit consideration of hypothetical voxel-wise FOV overlaps. The resulting depth predictions are accurate and agree on the omnidirectional multi-view geometry. Unlike existing MVS setups, MosaicMVS camera setup can be easily applied to omnidirectional indoor scenes without having to account for con-straints such as intricate epipolar constraints and the distortion of omnidirectional cameras. We validate the effectiveness of our framework on a new challenging indoor dataset in terms of depth estimation, reconstruction, and view synthesis. We also present new evaluation metric to check reconstruction performance using post-processed masks for accurate evaluation without any ground truth depth map or laser-scanned reconstructions. Experimental results show that our framework outperforms the state-of-the-art MVS methods in a large margin in all test scenes.

Scale-preserving shape reconstruction from monocular endoscope image sequences by supervised depth learning.

Reconstructing 3D shapes from images are becoming popular, but such methods usually estimate relative depth maps with ambiguous scales. A method for reconstructing a scale-preserving 3D shape from monocular endoscope image sequences through training an absolute depth prediction network is proposed. First, a dataset of synchronized sequences of RGB images and depth maps is created using an endoscope simulator. Then, a supervised depth prediction network is trained that estimates a depth map from a RGB image minimizing the loss compared to the ground-truth depth map. The predicted depth map sequence is aligned to reconstruct a 3D shape. Finally, the proposed method is applied to a real endoscope imagesequence.

Robust training for multi-view stereo networks with noisy labels

Recent multi-view stereo (MVS) works benefit a lot from deep neural networks with their great representation ability. However, top-performing networks require accurate depth ground truth rendered from scanned 3D models for training, which is quite challenging to capture in real-world scenarios. Thus, dealing with label noise contained in the dense depth labels is necessary for training when applied in complex real-world conditions, which has not been addressed before. In this paper, we investigate the effect of label noise in the ground truth depth maps by append noisy labels to the given accurate labels, and find that MVS networks are susceptible to noisy training data. To tackle this problem, we proposed a novel label noise-robust training framework for MVS networks. First, we propose a co-teaching based noise-robust training method to filter label noise contained in the dense depth ground truth, which trains two parallel network branches simultaneously and exchanges the noise selection capability to each other. Then, we present a co-training scheme that pseudo labels, the predicted depth maps that passes the geometric consistency check, are utilized as supervision to the peer network. Finally, the photometric consistency loss is introduced as additional supervision for all pixels. Experimental results demonstrate the robustness of deep MVS networks trained by our algorithm under low-level noise rate (40%) and high-level noise rate (65%), and the comparable 3D reconstruction performance to the network trained with label noise free datasets.

ActiveZero++: Mixed Domain Learning Stereo and Confidence-based Depth Completion with Zero Annotation.

Learning-based stereo methods usually require a large scale dataset with depth, however obtaining accurate depth in the real domain is difficult, but groundtruth depth is readily available in the simulation domain. In this paper we propose a new framework, ActiveZero++, which is a mixed domain learning solution for active stereovision systems that requires no real world depth annotation. In the simulation domain, we use a combination of supervised disparity loss and self-supervised loss on a shape primitives dataset. By contrast, in the real domain, we only use self-supervised loss on a dataset that is out-of-distribution from either training simulation data or test real data. To improve the robustness and accuracy of our reprojection loss in hard-to-perceive regions, our method introduces a novel self-supervised loss called temporal IR reprojection. Further, we propose the confidence-based depth completion module, which uses the confidence from the stereo network to identify and improve erroneous areas in depth prediction through depth-normal consistency. Extensive qualitative and quantitative evaluations on real-world data demonstrate state-of-the-art results that can even outperform a commercial depth sensor. Furthermore, our method can significantly narrow the Sim2Real domain gap of depth maps for state-of-the-art learning based 6D pose estimation algorithms.

Colonoscopy 3D video dataset with paired depth from 2D-3D registration.

Screening colonoscopy is an important clinical application for several 3D computer vision techniques, including depth estimation, surface reconstruction, and missing region detection. However, the development, evaluation, and comparison of these techniques in real colonoscopy videos remain largely qualitative due to the difficulty of acquiring ground truth data. In this work, we present a Colonoscopy 3D Video Dataset (C3VD) acquired with a high definition clinical colonoscope and high-fidelity colon models for benchmarking computer vision methods in colonoscopy. We introduce a novel multimodal 2D-3D registration technique to register optical video sequences with ground truth rendered views of a known 3D model. The different modalities are registered by transforming optical images to depth maps with a Generative Adversarial Network and aligning edge features with an evolutionary optimizer. This registration method achieves an average translation error of 0.321 millimeters and an average rotation error of 0.159 degrees in simulation experiments where error-free ground truth is available. The method also leverages video information, improving registration accuracy by 55.6% for translation and 60.4% for rotation compared to single frame registration. 22 short video sequences were registered to generate 10,015 total frames with paired ground truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models. The dataset also includes screening videos acquired by a gastroenterologist with paired ground truth pose and 3D surface models. The dataset and registration source code are available at https://durr.jhu.edu/C3VD.

Self-Supervised Monocular Depth Estimation With Self-Reference Distillation and Disparity Offset Refinement

Monocular depth estimation plays a fundamental role in computer vision. Due to the costly acquisition of depth ground truth, self-supervised methods that leverage adjacent frames to establish a supervision signal have emerged as the most promising paradigms. In this work, we propose two novel ideas to improve self-supervised monocular depth estimation: 1) self-reference distillation and 2) disparity offset refinement. Specifically, we use a parameter-optimized model as the teacher updated as the training epochs to provide additional supervision during the training process. The teacher model has the same structure as the student model, with weights inherited from the historical student model. In addition, a multiview check is introduced to filter out the outliers produced by the teacher model. Furthermore, we leverage the contextual consistency between high-level and low-level features to obtain multiscale disparity offsets, which are used to refine the disparity output incrementally by aligning disparity information at different scales. The experimental results on the KITTI and Make3D datasets show that our method outperforms previous state-of-the-art competitors.

A Method for Training Object Scale Estimation System using Feature Extraction Enhancement with Depth Estimation

In recent years, machine learning-based object scale estimation has been growing in popularity, as the significance of the technology lies in its potential for use in many industry fields. Although several methods have been proposed, the possible applications of this technique are limited due to its insufficient accuracy. Hence, a human-level accurate system is needed for the technology to be applied in the real-world domain. This research paper proposes a novel object scale estimation system that incorporates the feature extractor, disentangled feature maps, depth estimator, object localizer, and ground truth depth map. The input of the proposed system is an image, which is inputted into the feature extractor to create disentangled feature maps. These feature maps are then extracted by the depth estimator to generate a depth map, and by the object localizer to create a predicted bounding box around the object. The trained feature extractor can extract disentangled size-related features from the inputted image by jointly training the depth estimator and object localizer. The use of disentangled features boosts the performance of the proposed system. In addition, we propose an actual scale converter module to calculate the actual size of the inputted object. Throughout the experiments, the proposed method has proven that it is superior compared to other state-of-the-art methods. The proposed method achieves an IoU (Intersection over Union) value of 0.8113 on the COCO dataset.&#x0D;

Multi-resolution distillation for self-supervised monocular depth estimation

Obtaining dense depth ground-truth is not trivial, which leads to the introduction of self-supervised monocular depth estimation. Most self-supervised methods utilize the photometric loss as the primary supervisory signal to optimize a depth network. However, such self-supervised training often falls into an undesirable local minimum due to the ambiguity of the photometric loss. In this paper, we propose a novel self-distillation training scheme that provides a new self-supervision signal, depth consistency among different input resolutions, to the depth network. We further introduce a gradient masking strategy that adjusts the self-supervision signal of the depth consistency during back-propagation to boost the effectiveness of our depth consistency. Experiments demonstrate that our method brings meaningful performance improvements when applied to various depth network architectures. Furthermore, our method outperforms the existing self-supervised methods on KITTI, Cityscapes, and DrivingStereo datasets by a noteworthy margin.