Accurate Depth Map Research Articles

An Efficient Dense Reconstruction Algorithm from LiDAR and Monocular Camera

Dense reconstruction have been studied for decades in the fields of computer vision and robotics, in which LiDAR and camera are widely used. However, vision-based methods are sensitive to illumination variation and lack direct depth, and LiDAR-based methods are limited by sparse LiDAR measurement and lacking color and texture information. In this paper, we propose a novel 3D reconstruction algorithm based on LiDAR and a monocular camera, which realizes dense reconstruction. In the algorithm, a LiDAR odometry is used to get accurate poses and poses calculated by the odometry module are used in the calculation of depth maps and fusion of depth maps, and then mesh and texture mapping are implemented. In addition, a semantic segmentation network and a depth completion network are used to obtain dense and accurate depth maps. The concept of symmetry is utilized to generate 3D models of objects or scenes; that is, the reconstruction and camera imaging of these objects or scenes are symmetrical. Experimental results on public dataset show that the proposed algorithm achieves higher accuracy, efficiency and completeness than existing methods.

CAFNet: Context aligned fusion for depth completion

Depth completion aims at reconstructing a dense depth from sparse depth input, frequently using color images as guidance. The sparse depth map lacks sufficient contexts for reconstructing focal contexts such as the shape of objects. The RGB images contain redundant contexts including details useless for reconstruction, which reduces the efficiency of focal context extraction. The unaligned contextual information from these two modalities poses a challenge to focal context extraction and further fusion, as well as the accuracy of depth completion. To optimize the utilization of multimodal contextual information, we explore a novel framework: Context Aligned Fusion Network (CAFNet). CAFNet comprises two stages: the context-aligned stage and the full-scale stage. In the context-aligned stage, CAFNet downsamples input RGB-D pairs to the scale, at which multimodal contextual information is adequately aligned for feature extraction in two encoders and fusion in CF modules. In the full-scale stage, feature maps with fused multimodal context from the previous stage are upsampled to the original scale and subsequentially fused with full-scale depth features by the GF module utilizing a dynamic masked fusion strategy. Ultimately, accurate dense depth maps are reconstructed, leveraging the GF module’s resultant features. Experiments conducted on indoor and outdoor benchmark datasets show that the CAFNet produces results comparable to state-of-the-art methods while effectively reducing computational costs.

Employment of conditional random fields for monocular depth estimation

Abstract Estimating depth from a solitary RGB image, known as monocular depth estimation, presents a significant challenge. Currently, most methods for this task involve designing increasingly complex networks to regress the depth map straightforwardly. However, we have adopted a more interpretable approach by using Conditional Random Fields from optimization methods. Additionally, to facilitate better information transfer between nodes, a multi-head attention mechanism is employed to calculate multiple energy functions, which are then optimized by the network into an accurate depth map. Experiments demonstrate that our method can accurately estimate the depth of landscapes.

A vision measurement method for ship hull plates based on multi-view stereo and image segmentation

The background denoising of point clouds is challenging for the measurement of ship hull plates. To address this issue, a vision measurement method is proposed by combining an identity cost volume network (ICV-Net) and a segment anything model (SAM). By applying the ship hull plate masks segmented by SAM to the corresponding depth maps inferred by ICV-Net, the background noise can be directly removed at the depth map level. Additionally, an edge extraction optimization algorithm complementary to SAM is proposed to eliminate the thickness-related noise. Moreover, an efficient depth map refinement algorithm is proposed by imposing the smoothness prior of ship hull plate surfaces and multi-view photometric consistency. Sub-pixel accurate depth maps of ship hull plates can be obtained by the depth map refinement. Experimental results suggest that the proposed method can automatically, accurately and efficiently remove the background noise of ship hull plates, which is crucial toward practical application.

SLAM for Indoor Mapping of Wide Area Construction Environments

Abstract. Simultaneous localization and mapping (SLAM), i.e., the reconstruction of the environment represented by a (3D) map and the concurrent pose estimation, has made astonishing progress. Meanwhile, large scale applications aiming at the data collection in complex environments like factory halls or construction sites are becoming feasible. However, in contrast to small scale scenarios with building interiors separated to single rooms, shop floors or construction areas require measures at larger distances in potentially texture less areas under difficult illumination. Pose estimation is further aggravated since no GNSS measures are available as it is usual for such indoor applications. In our work, we realize data collection in a large factory hall by a robot system equipped with four stereo cameras as well as a 3D laser scanner. We apply our state-of-the-art LiDAR and visual SLAM approaches and discuss the respective pros and cons of the different sensor types for trajectory estimation and dense map generation in such an environment. Additionally, dense and accurate depth maps are generated by 3D Gaussian splatting, which we plan to use in the context of our project aiming on the automatic construction and site monitoring.

DepthGAN: GAN-based depth generation from semantic layouts

AbstractExisting GAN-based generative methods are typically used for semantic image synthesis. We pose the question of whether GAN-based architectures can generate plausible depth maps and find that existing methods have difficulty in generating depth maps which reasonably represent 3D scene structure due to the lack of global geometric correlations. Thus, we propose DepthGAN, a novel method of generating a depth map using a semantic layout as input to aid construction, and manipulation of well-structured 3D scene point clouds. Specifically, we first build a feature generation model with a cascade of semantically-aware transformer blocks to obtain depth features with global structural information. For our semantically aware transformer block, we propose a mixed attention module and a semantically aware layer normalization module to better exploit semantic consistency for depth features generation. Moreover, we present a novel semantically weighted depth synthesis module, which generates adaptive depth intervals for the current scene. We generate the final depth map by using a weighted combination of semantically aware depth weights for different depth ranges. In this manner, we obtain a more accurate depth map. Extensive experiments on indoor and outdoor datasets demonstrate that DepthGAN achieves superior results both quantitatively and visually for the depth generation task.

DFNet-Trans: An end-to-end multibranching network for depth estimation for transparent objects

Transparent objects play a vital role in modern industries and find widespread applications across various engineering scenarios. However, capturing accurate depth maps of transparent objects remains challenging due to their reflective and refractive properties, which pose difficulties for most commercial-grade optical sensors. In this paper, we propose a novel depth estimation method called DFNet-Trans, designed to estimate depth from a noisy RGB-D image input. Initially, a multiscale feature fusion module (FFM) is incorporated into the existing depth estimation network to generate the initial depth map. Subsequently, we enhance the network by adding a confidence branch and a mask branch on the same encoder, enabling improved distortion correction and real scene restoration in the depth estimation. Based on the framework representation, missing depth can be completed. Comprehensive experiments demonstrate that the proposed approach significantly outperforms the current state-of-the-art methods on the recently popular large-scale real dataset TransCG. the proposed approach achieves a remarkable 27.7% reduction in RMSE and a notable 34.6% reduction in REL. The generalization experiment shows that the proposed approach outperforms existing methods when generalized to an unknown real dataset.

SC-DepthV3: Robust Self-Supervised Monocular Depth Estimation for Dynamic Scenes.

Self-supervised monocular depth estimation has shown impressive results in static scenes. It relies on the multi-view consistency assumption for training networks, however, that is violated in dynamic object regions and occlusions. Consequently, existing methods show poor accuracy in dynamic scenes, and the estimated depth map is blurred at object boundaries because they are usually occluded in other training views. In this paper, we propose SC-DepthV3 for addressing the challenges. Specifically, we introduce an external pretrained monocular depth estimation model for generating single-image depth prior, namely pseudo-depth, based on which we propose novel losses to boost self-supervised training. As a result, our model can predict sharp and accurate depth maps, even when training from monocular videos of highly dynamic scenes. We demonstrate the significantly superior performance of our method over previous methods on six challenging datasets, and we provide detailed ablation studies for the proposed terms.

Directional Ring Difference Filter for Robust Shape-from-Focus

In the shape-from-focus (SFF) method, the quality of the 3D shape generated relies heavily on the focus measure operator (FM) used. Unfortunately, most FMs are sensitive to noise and provide inaccurate depth maps. Among recent FMs, the ring difference filter (RDF) has demonstrated excellent robustness against noise and reasonable performance in computing accurate depth maps. However, it also suffers from the response cancellation problem (RCP) encountered in multidimensional kernel-based FMs. To address this issue, we propose an effective and robust FM called the directional ring difference filter (DRDF). In DRDF, the focus quality is computed by aggregating responses of RDF from multiple kernels in different directions. We conducted experiments using synthetic and real image datasets and found that the proposed DRDF method outperforms traditional FMs in terms of noise handling and producing a higher quality 3D shape estimate of the object.

Monocular Depth Estimation Using Res-UNet with an Attention Model

Depth maps are single image metrics that carry the information of a scene in three-dimensional axes. Accurate depth maps can recreate the 3D structure of a scene, which helps in understanding the full geometry of the objects within the scene. Depth maps can be generated from a single image or multiple images. Single-image depth mapping is also known as monocular depth mapping. Depth maps are ill-posed problems that are complex and require extensive calibration. Therefore, recent methods use deep learning to develop depth maps. We propose a new method in monocular depth estimation to develop a high-quality depth map. Our approach is based on a convolutional neural network in which we used Res-UNet with a spatial attention model to develop depth maps. The addition of an attention mechanism increases the capability of feature extraction and enhances the boundaries features. It does not add any extra parameters to the network. With our proposed model, we demonstrate that a simple CNN model aided with an attention mechanism can create high-quality depth maps with a small iteration and training time. Our model performs very well compared to the existing state-of-the-art methods on the benchmark NYU-depth v2 dataset. Our model is flexible and can be applied to any depth mapping or multi-segmentation tasks.

Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted Display.

This work introduces a perspective-corrected video see-through mixed-reality head-mounted display with edge-preserving occlusion and low-latency capabilities. To realize the consistent spatial and temporal composition of a captured real world containing virtual objects, we perform three essential tasks: 1) to reconstruct captured images so as to match the user's view; 2) to occlude virtual objects with nearer real objects, to provide users with correct depth cues; and 3) to reproject the virtual and captured scenes to be matched and to keep up with users' head motions. Captured image reconstruction and occlusion-mask generation require dense and accurate depth maps. However, estimating these maps is computationally difficult, which results in longer latencies. To obtain an acceptable balance between spatial consistency and low latency, we rapidly generated depth maps by focusing on edge smoothness and disocclusion (instead of fully accurate maps), to shorten the processing time. Our algorithm refines edges via a hybrid method involving infrared masks and color-guided filters, and it fills disocclusions using temporally cached depth maps. Our system combines these algorithms in a two-phase temporal warping architecture based upon synchronized camera pairs and displays. The first phase of warping is to reduce registration errors between the virtual and captured scenes. The second is to present virtual and captured scenes that correspond with the user's head motion. We implemented these methods on our wearable prototype and performed end-to-end measurements of its accuracy and latency. We achieved an acceptable latency due to head motion (less than 4 ms) and spatial accuracy (less than 0.1° in size and less than 0.3° in position) in our test environment. We anticipate that this work will help improve the realism of mixed reality systems.

Depth prediction by using various velocity models of Khasib Reservoir in East Baghdad field, Iraq

In this study, the variance seismic attribute was used to track fault geomorphology, which interpreted in time. To convert these fault surfaces and isochron maps to depth, the velocity model was prepared. Therefore, 14 velocity models for depth conversion were built by using different equations in East Baghdad Field, central Iraq to depth conversion assessment and select a precise model from many candidates of velocity models to generate an accurate depth map. All the velocity models are divided into two main velocity layers. The upper-velocity layer represents the tertiary sequences, while the lower-velocity layer represents the Upper Cretaceous sequences. Three criteria are made in this study (mis-tie value, standard deviation analysis, and blind well test) to select the best depth map. The results reveal the velocity model-7 is better to use because the depth map of the Khasib Reservoir, which produced from model-7 shows less mis-tie between estimated and real depth, less standard deviation values, and lower errors at EB 24 and 26 well locations. The corrected depth map of Khasib Reservoir shows three structure noses. Two of them opened toward the NW direction, while the southern structural nose opened toward the SW direction with a minor enclosure.

Multistage Pixel-Visibility Learning With Cost Regularization for Multiview Stereo

Multiple-view stereo has potential applications in robotic operations and autonomous driving (unstructured environment construction, visual servo). With assisted depth information, inertial navigation systems can achieve precise navigation. It is, especially suitable for GPS failures in complex environments. Accurate depth estimation is a challenge in low-textured or occluded regions. To alleviate the inference of incorrect depth, a multi-stage pixel-visibility learning-based stereo network is presented in this paper. Its improvements are as follows: 1) a new content-adaptive cost volume aggregation mechanism based on neighboring pixel-wise visibility is designed to effectively produce more accurate and smoother depth map predictions in the object boundary. 2) global convolution block and boundary refinement block are developed to regularize its cost volume, they can learn the inherent constraints of feature matching correspondence and effectively mitigate the depth estimation uncertainty in low-textured regions. 3) a new loss function is designed to measure the uncertainty of predicted probability distribution and enhance the reliability of depth map inference. Experimental results on the indoor DTU datasets and the outdoor Tanks & Temples datasets indicate that our method can achieve superior performance and has a powerful generalization ability, which is comparable to state-of-the-art works. Note to Practitioners—Multiple-view stereo (MVS) can estimate dense 3D representations of scenes, which is widely used in autonomous driving, robotic navigation, virtual reality (VR), and augmented reality (AR). Aiming at the problem of incorrect depth inference in low-textured or occluded regions, this work proposes a novel multi-stage depth prediction method based on neighboring pixel-wise visibility. Our method cannot only achieve accurate depth estimation for robot perception but also make no concession to real-time performance. It is clear that the proposed method has good potential in 3D reconstruction, robotic navigation, and VR/AR fields to provide accurate depth estimation in real-time with limited memory consumption.

Self-Supervised Monocular Depth Estimation With Geometric Prior and Pixel-Level Sensitivity

Self-supervised monocular depth estimation has gained popularity due to its convenience of training network without dense ground truth depth annotation. Specifically, the multi-frame monocular depth estimation achieves promising results in virtue of temporal information. However, existing multi-frame solutions ignore the different impacts of pixels of input frame on depth estimation and the geometric information is still insufficiently explored. In this paper, a self-supervised monocular depth estimation framework with geometric prior and pixel-level sensitivity is proposed. Geometric constraint is involved through a geometric pose estimator with prior depth predictor and optical flow predictor. Further, an alternative learning strategy is designed to improve the learning of prior depth predictor by decoupling it with the ego-motion from the geometric pose estimator. On this basis, prior feature consistency regularization is introduced into the depth encoder. By taking the dense prior cost volume based on optical flow map and ego-motion as the supervising signal for feature consistency learning, the cost volume is obtained with more reasonable feature matching. To deal with the pixel-level difference of sensitivity in input frame, a sensitivity-adaptive depth decoder is built by flexibly adding a shorter path from cost volume to the final depth prediction. In this way, the back propagation of gradient to cost volume is adaptively adjusted, and an accurate depth map is decoded. The effectiveness of the proposed method is verified on public datasets.

High-Accuracy Depth Estimation of Nematic Liquid Crystal Microlens Arrays Based on Convex Optimization Theory

We have proposed an iterative feedback method based on a nematic liquid crystal microlens array (LC-MLA) light-field system, which can obtain highly accurate depth maps. A preliminary reconstruction of the all-in-focus map of the scene is produced to obtain a high-precision depth map. A natural image sparsity prior and a chromatic aberration correction are used to ensure high-quality deblurring in the presence of inaccurate depth estimation. Subsequently, a high-quality all-in-focus image is used as a constraint to obtain a high-accuracy depth estimation result. After the experiment, it can be demonstrated that the proposed method can provide good estimation results under texture-free conditions.

Towards Comprehensive Monocular Depth Estimation: Multiple Heads are Better Than One

Depth estimation attracts widespread attention in the computer vision community. However, it is still quite difficult to recover an accurate depth map using only one RGB image. We observe a phenomenon that existing methods tend to fail in different cases, caused by differences in network architecture, loss function and so on. In this work, we investigate into the phenomenon and propose to integrate the strengths of multiple weak depth predictor to build a comprehensive and accurate depth predictor, which is critical for many real-world applications, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , 3D reconstruction. Specifically, we construct multiple base (weak) depth predictors by utilizing different Transformer-based and convolutional neural network (CNN)-based architectures. Transformer establishes long-range correlation while CNN preserves local information ignored by Transformer due to the spatial inductive bias. Therefore, the coupling of Transformer and CNN contributes to the generation of complementary depth estimates, which are essential to achieve a comprehensive depth predictor. Then, we design mixers to learn from multiple weak predictions and adaptively fuse them into a strong depth estimate. The resultant model, which we refer to as Transformer-assisted depth ensembles (TEDepth). On the standard NYU-Depth-v2 and KITTI datasets, we thoroughly explore how the neural ensembles affect the depth estimation and demonstrate that our TEDepth achieves better results than previous state-of-the-art approaches. To validate the generalizability across cameras, we directly apply the models trained on NYU-Depth-v2 to the SUN RGB-D dataset without any fine-tuning, and the superior results emphasize its strong generalizability. The source code and trained models will be publicly available upon the acceptance.

Depth Map Prediction of Occluded Objects Using Structure Tensor with Gain Regularization

The creation of the 3D rendering model involves the prediction of an accurate depth map for the input images. A proposed approach of a modified semi-global block matching algorithm with variable window size and the gradient assessment of objects predicts the depth map. 3D modeling and view synthesis algorithms could effectively handle the obtained disparity maps. This work uses the consistency check method to find an accurate depth map for identifying occluded pixels. The prediction of the disparity map by semi-global block matching has used the benchmark dataset of Middlebury stereo for evaluation. The improved depth map quality within a reasonable processing time outperforms the other existing depth map prediction algorithms. The experimental results have shown that the proposed depth map predictioncould identify the inter-object boundaryeven with the presence ofocclusion with less detection error and runtime.We observed that the Middlebury stereo dataset has very few images with occluded objects, which made the attainment of gain cumbersome. Considering this gain, we have created our dataset with occlusion using the structured lighting technique. The proposed regularization term as an optimization process in the graph cut algorithm handles occlusion for different smoothing coefficients. The experimented results demonstrated that our dataset had outperformed the Tsukuba dataset regarding the percentage of occluded pixels.

Guided Depth Completion with Instance Segmentation Fusion in Autonomous Driving Applications

Pixel-level depth information is crucial to many applications, such as autonomous driving, robotics navigation, 3D scene reconstruction, and augmented reality. However, depth information, which is usually acquired by sensors such as LiDAR, is sparse. Depth completion is a process that predicts missing pixels' depth information from a set of sparse depth measurements. Most of the ongoing research applies deep neural networks on the entire sparse depth map and camera scene without utilizing any information about the available objects, which results in more complex and resource-demanding networks. In this work, we propose to use image instance segmentation to detect objects of interest with pixel-level locations, along with sparse depth data, to support depth completion. The framework utilizes a two-branch encoder-decoder deep neural network. It fuses information about scene available objects, such as objects' type and pixel-level location, LiDAR, and RGB camera, to predict dense accurate depth maps. Experimental results on the KITTI dataset showed faster training and improved prediction accuracy. The proposed method reaches a convergence state faster and surpasses the baseline model in all evaluation metrics.

Cross-validation of time-depth conversion and evaluation of different approaches in the Mesopotamian Basin, Iraq

The time-depth conversion process is a significant task in seismic interpretation to establish the link between geophysical information in the time domain and geological information in the depth domain at/away from well locations. Selecting the suitable velocity model for time-depth conversion to generate an accurate depth map is difficult if the accuracy of these models is unknown. In the current study, the cross-validation technique is used as a tool to diagnose and evaluate the performance of time-depth conversion at/away from well controls to predict the depth of Top Hartha and Zubair reservoirs using the dataset of East Baghdad Oil Field. To test this technique, four common velocity model approaches used for time-depth conversion with different scenarios of velocity parameters (initial velocity V 0 and depth gradient (K)) were applied to produce ten velocity models (1–10). According to the gradient variation of velocity with depth, check shot analysis, the velocity models (1–10) include three key velocities layer-cakes: Layer 1 (Middle Miocene-Upper Cretaceous), Layer 2 (Upper Cretaceous), and Layer 3 (Lower Cretaceous) with 18 horizons from Middle Miocene down to Lower Cretaceous. The cross-validation analysis reveals that the velocity model with a variable surface initial velocity and constant depth gradient (Model 9) was the most accurate with fewer mistie between actual and predicted depth. Consequently, this model is used to construct the depth map of the Hartha and Zubair reservoirs. Finally, this study progresses a workflow that can be applied to the region with any geological setting to investigate time-depth conversion uncertainty.

An integrated imaging sensor for aberration-corrected 3D photography

Planar digital image sensors facilitate broad applications in a wide range of areas1–5, and the number of pixels has scaled up rapidly in recent years2,6. However, the practical performance of imaging systems is fundamentally limited by spatially nonuniform optical aberrations originating from imperfect lenses or environmental disturbances7,8. Here we propose an integrated scanning light-field imaging sensor, termed a meta-imaging sensor, to achieve high-speed aberration-corrected three-dimensional photography for universal applications without additional hardware modifications. Instead of directly detecting a two-dimensional intensity projection, the meta-imaging sensor captures extra-fine four-dimensional light-field distributions through a vibrating coded microlens array, enabling flexible and precise synthesis of complex-field-modulated images in post-processing. Using the sensor, we achieve high-performance photography up to a gigapixel with a single spherical lens without a data prior, leading to orders-of-magnitude reductions in system capacity and costs for optical imaging. Even in the presence of dynamic atmosphere turbulence, the meta-imaging sensor enables multisite aberration correction across 1,000 arcseconds on an 80-centimetre ground-based telescope without reducing the acquisition speed, paving the way for high-resolution synoptic sky surveys. Moreover, high-density accurate depth maps can be retrieved simultaneously, facilitating diverse applications from autonomous driving to industrial inspections.