Depth Estimation Research Articles

Structure tensor-based Gaussian kernel edge-adaptive depth map refinement with triangular point view in images

<span lang="EN-US">Image reconstruction is the process of restoring the image resolution. In 3D image reconstruction, the objects in different viewpoints are processed with the triangular point view (TPV) method to estimate object geometry structure for 3D model. This work proposes a depth refinement methodology in preserving the geometric structure of objects using the structure tensor method with a Gaussian filter by transforming a series of 2D input images into a 3D model. The computation of depth map errors can be found by comparing the masked area/patch with the distribution of the original image's greyscale levels using the error pixel-based patch extraction algorithm. The presence of errors in the depth estimation could seriously deteriorate the quality of the 3D effect. The depth maps were iteratively refined based on histogram bins number to improve the accuracy of initial depth maps reconstructed from rigid objects. The existing datasets such as the dataset tanks and unit (DTU) and Middlebury datasets, were used to build the model out of the object scene structure. The results of this work have demonstrated that the proposed patch analysis outperformed the existing state of the art models depth refinement methods in terms of accuracy.</span>

Depth-Enhancement Network for Monocular 3D object detection

Abstract To accurately obtain 3D information, the correct use of depth data is crucial. Compared with radar-based methods, detecting objects in 3D space in a single image is extremely challenging due to the lack of depth cues. However, monocular 3D object detection provides a more economical solution. Traditional monocular 3D object detection methods often rely on geometric constraints, such as key points, object shape relationships, and 3D to 2D optimization, to address the inherent lack of depth information. However, these methods still make it challenging to extract rich information directly from depth estimation for fusion. To fundamentally enhance the ability of monocular 3D object detection, we propose a monocular 3D object detection network based on depth information enhancement. The network learns object detection and depth estimation tasks simultaneously through a unified framework, integrates depth features as auxiliary information into the detection branch, and then constrains and enhances them to obtain better spatial representation. To this end, we introduce a new cross-modal fusion strategy, which realizes a more reasonable fusion of cross-modal information by exploring redundant, complementary information and their interactions between RGB features and depth features. Extensive experiments on the KITTI dataset show that our method can significantly improve the performance of monocular 3D object detection.

Implementation Paper on Automated 2D Image to 3D Model Generation

With the rise of digital world, traditional learning methods, while time-tested, often fall short in meeting the dynamic educational needs of today’s learners. They are constrained by several drawbacks that impede effective comprehension and engagement. This abstract sheds light on the limitations of traditional learning approaches and introduces how the integration of 3D model reconstruction can mitigate these challenges, thereby enhancing the learning experience. We proposed deep Conventional techniques, which create a virtual view from a reference picture, use two distinct steps: depth image-based rendering (DIBR) with estimated depth and depth (or disparity) estimation for the reference image and based on these steps we can create 3D model. Which allows educators to create immersive and interactive learning environments that engage students on a profound level. By converting abstract concepts into tangible and visually stimulating representations, 3D models bridge the gap between theory and practice, making complex subjects more accessible. Keywords: 2D Image, 3D Model, CNN, DIBR

Automated floodwater depth estimation using large multimodal model for rapid flood mapping

Automated floodwater depth estimation using large multimodal model for rapid flood mapping

Estimation of Curie-point depth, heat flow and geothermal gradients in Shelleng area and environs, northeastern Nigeria from aeromagnetic data

Estimation of Curie-point depth, heat flow and geothermal gradients in Shelleng area and environs, northeastern Nigeria from aeromagnetic data

Calibration-free deep optics for depth estimation with precise simulation

Monocular depth estimation is an important computer vision task widely explored in fields such as autonomous driving, robotics, and more. Recently, deep optics approaches that optimize diffractive optical elements (DOEs) with differentiable frameworks have improved depth estimation performance. However, they only consider on-axis point spread functions (PSFs) and are highly dependent on system calibration. We propose a precise end-to-end paradigm combining ray tracing and angular spectrum diffraction. With this approach, we jointly trained a DOE and a reconstruction network for depth estimation and image restoration. Compared with conventional deep optics approaches, we accurately simulate both on-axis and off-axis PSFs, eliminating the need for calibration. We have validated the high similarity between captured PSFs and simulated ones at 19.4∘ field-of-view (FOV). Our optimized phase mask and network achieve state-of-the-art performance among semantic-based monocular depth estimation and existing deep optics methods. During real-world experiments, our prototype camera shows depth distributions similar to the Intel D455 infrared structured light camera.

Single-photon lidar depth imaging based on sparsity adaptive matching pursuit method

Photon counting lidar has the ability to detect weak echo photons. However, in imaging applications, a small number of echo signals obtained in a very short acquisition time cannot get an accurate image estimation through the photon arrival time distribution histogram. Therefore, high-precision and fast imaging of an object using a small amount of weak echo photon signals is urgent to be solved. In this work, a sparse adaptive matching pursuit (SAMP) algorithm is proposed to process the echo photons without prior information to obtain the effective photon interval, which improves the reconstruction quality and imaging speed. By constructing a non-coaxial scanning imaging system to detect the face model, it has been experimentally proven that the proposed algorithm can achieve depth and reflectance image estimation under the condition of low power and low acquisition time of 4µs per pixel on average, and the mean absolute error (MAE) value is reduced by about 10 times compared to conventional reconstruction algorithm.

Implementing robust outlier detection to enhance estimation accuracy of GNSS-IR based seasonal snow depth retrievals

ABSTRACT Monitoring snow depth variations aligned with the seasonal cycle is crucial for studies on climate change and its impacts due to global warming. Especially in terms of snow hydrology, the determination and continuous monitoring of snow depth to determine snow water equivalent is a priority in climate, water science, drought, flood, and inundation prediction studies, particularly in the water cycle. Recently, the so-called Global Navigation Satellite Systems – Interferometric Reflectometry (GNSS-IR) method enables to extract environmental radiometric and geometric characteristics of surface where the signals transmitted from the satellites reflect. The reflected signal follows an extra path than the direct one and causes a major error source for accurate point positioning but a useful tool for sensing the environment. However, outliers remain a challenge in long-term GNSS-IR estimations. In this study, we proposed a median-based robust outlier detection (ROD) approach for identifying outliers in long-term GNSS-IR snow depth estimations. To validate our approach, we analysed 5-year GNSS L1 SNR and L2 SNR data from 1 January 2015 to 31 December 2019 provided by AB33 and AB39 GNSS stations in Alaska, U.S.A. using the GNSS-IR method. We validated GNSS-IR estimations using snow depth measurements from the Coldfoot and Fort Yukon stations in the SNOTEL network. Applying ROD to long-term snow depth estimates increased the highest correlation from 91.58% to 94.93%, and reduced the lowest RMSE from 8.6 cm to 6.7 cm. In addition, the improvement rates calculated to assess the contribution of ROD to the results showed improvements of up to 6.9% in correlation and 26.1% in RMSE. Overall, the results demonstrate that ROD can be effectively used to detect outliers in long-term GNSS-IR snow depth time series.

An Image-Based Interactive Training Method of an Upper Limb Rehabilitation Robot

Aimed at the problem of human–machine interaction between patients and robots in the process of using rehabilitation robots for rehabilitation training, this paper proposes a human–machine interactive control method based on an independently developed upper limb rehabilitation robot. In this method, the camera is used as a sensor, the human skeleton model is used to analyse the moving image, and the key points of the human body are extracted. Then, the three-dimensional coordinates of the key points of the human arm are extracted by depth estimation and spatial geometry, and then the real-time motion data are obtained, and the control instructions of the robot are generated from it to realise the real-time interactive control of the robot. This method can not only improve the adaptability of the system to individual patient differences, but also improve the robustness of the system, which is less affected by environmental changes. The experimental results show that this method can realise real-time control of the rehabilitation robot, and that the robot assists the patient to complete the action with high accuracy. The results show that this control method is effective and can be applied to the fields of robot control and robot-assisted rehabilitation training.

An endoscopic chisel: intraoperative imaging carves 3D anatomical models.

Preoperative imaging plays a pivotal role in sinus surgery where CTs offer patient-specific insights of complex anatomy, enabling real-time intraoperative navigation to complement endoscopy imaging. However, surgery elicits anatomical changes not represented in the preoperative model, generating an inaccurate basis for navigation during surgery progression. We propose a first vision-based approach to update the preoperative 3D anatomical model leveraging intraoperative endoscopic video for navigated sinus surgery where relative camera poses are known. We rely on comparisons of intraoperative monocular depth estimates and preoperative depth renders to identify modified regions. The new depths are integrated in these regions through volumetric fusion in a truncated signed distance function representation to generate an intraoperative 3D model that reflects tissue manipulation RESULTS: We quantitatively evaluate our approach by sequentially updating models for a five-step surgical progression in an ex vivo specimen. We compute the error between correspondences from the updated model and ground-truth intraoperative CT in the region of anatomical modification. The resulting models show a decrease in error during surgical progression as opposed to increasing when no update is employed. Our findings suggest that preoperative 3D anatomical models can be updated using intraoperative endoscopy video in navigated sinus surgery. Future work will investigate improvements to monocular depth estimation as well as removing the need for external navigation systems. The resulting ability to continuously update the patient model may provide surgeons with a more precise understanding of the current anatomical state and paves the way toward a digital twin paradigm for sinus surgery.

Image Intrinsic-Based Unsupervised Monocular Depth Estimation in Endoscopy.

Unsupervised monocular depth estimation plays a vital role for endoscopy-based minimally invasive surgery (MIS). However, it remains challenging due to the distinctive imaging characteristics of endoscopy which disrupt the assumption of photometric consistency, a foundation relied upon by conventional methods. Distinct from recent approaches taking image pre-processing strategy, this paper introduces a pioneering solution through intrinsic image decomposition (IID) theory. Specifically, we propose a novel end-to-end intrinsic-based unsupervised monocular depth learning framework that is comprised of an image intrinsic decomposition module and a synthesis reconstruction module. This framework seamlessly integrates IID with unsupervised monocular depth estimation, and dedicated losses are meticulously designed to offer robust supervision for network training based on this novel integration. Noteworthy, we rely on the favorable property of the resulting albedo map of IID to circumvent the challenging images characteristics instead of pre-processing the input frames. The proposed method is extensively validated on SCARED and Hamlyn datasets, and better results are obtained than state-of-the-art techniques. Beside, its generalization ability and the effectiveness of the proposed components are also validated. This innovative method has the potential to elevate the quality of 3D reconstruction in monocular endoscopy, thereby enhancing the accuracy and robustness of augmented reality navigation technology in MIS. Our code will be available at: https://github.com/bobo909/IID-SfmLearner.

Mean Texture Depth Estimation of Exposed Aggregate Concrete Pavement Surface Texture Based on Photogrammetry Technique

Mean Texture Depth Estimation of Exposed Aggregate Concrete Pavement Surface Texture Based on Photogrammetry Technique

Single-hydrophone-based passive localization for vertically moving targets in shallow water

In this article, we consider underwater passive localization for vertically moving targets in shallow water. A method that utilizes parameters derived from interference structures is introduced, which enables the implementation of both range and depth estimations for targets with a single hydrophone. Underwater moving targets radiate broadband continuous spectrum signals by forming multipath signals through interface reflections, resulting in stable interference structures in the time-frequency domain of the received signals. Ray theory is utilized to derive the trajectory equations of the interference fringes. Considering the periodic characteristics of interference structures, a specialized homomorphic filter is designed to effectively restore the peaks of the fringes to reduce localization errors. Furthermore, with a priori information on the target's initial depth, the Hough Transform is used to extract the target's location information within interference fringes formed by direct signal and surface-reflected signal. Addressing the issue of multi-peaks in parameter domain, parameter information across multiple fringes is fused. The effectiveness and robustness of the proposed method have been demonstrated through numerical simulations, lake trial, and error assessments, even when underwater environments information is not precisely known.

Event-based depth estimation with dense occlusion

Event-based depth estimation with dense occlusion

A Novel Lightweight Model for Underwater Image Enhancement.

Underwater images suffer from low contrast and color distortion. In order to improve the quality of underwater images and reduce storage and computational resources, this paper proposes a lightweight model Rep-UWnet to enhance underwater images. The model consists of a fully connected convolutional network and three densely connected RepConv blocks in series, with the input images connected to the output of each block with a Skip connection. First, the original underwater image is subjected to feature extraction by the SimSPPF module and is processed through feature summation with the original one to be produced as the input image. Then, the first convolutional layer with a kernel size of 3 × 3, generates 64 feature maps, and the multi-scale hybrid convolutional attention module enhances the useful features by reweighting the features of different channels. Second, three RepConv blocks are connected to reduce the number of parameters in extracting features and increase the test speed. Finally, a convolutional layer with 3 kernels generates enhanced underwater images. Our method reduces the number of parameters from 2.7 M to 0.45 M (around 83% reduction) but outperforms state-of-the-art algorithms by extensive experiments. Furthermore, we demonstrate our Rep-UWnet effectively improves high-level vision tasks like edge detection and single image depth estimation. This method not only surpasses the contrast method in objective quality, but also significantly improves the contrast, colorimetry, and clarity of underwater images in subjective quality.

Unsupervised Domain Adaptation Depth Estimation Based on Self-attention Mechanism and Edge Consistency Constraints

In the unsupervised domain adaptation (UDA) (Akada et al. Self-supervised learning of domain invariant features for depth estimation, in: 2022 IEEE/CVF winter conference on applications of computer vision (WACV), pp 3377–3387 (2022). 10.1109/WACV51458.2022.00107) depth estimation task, a new adaptive approach is to use the bidirectional transformation network to transfer the style between the target and source domain inputs, and then train the depth estimation network in their respective domains. However, the domain adaptation process and the style transfer may result in defects and biases, often leading to depth holes and instance edge depth missing in the target domain’s depth output. To address these issues, We propose a training network that has been improved in terms of model structure and supervision constraints. First, we introduce a edge-guided self-attention mechanism in the task network of each domain to enhance the network’s attention to high-frequency edge features, maintain clear boundaries and fill in missing areas of depth. Furthermore, we utilize an edge detection algorithm to extract edge features from the input of the target domain. Then we establish edge consistency constraints between inter-domain entities in order to narrow the gap between domains and make domain-to-domain transfers easier. Our experimental demonstrate that our proposed method effectively solve the aforementioned problem, resulting in a higher quality depth map and outperforming existing state-of-the-art methods.

Self-Supervised Monocular Depth Estimation via Binocular Geometric Correlation Learning

Monocular depth estimation aims to infer a depth map from a single image. Although supervised learning-based methods have achieved remarkable performance, they generally rely on a large amount of labor-intensively annotated data. Self-supervised methods on the other hand do not require any annotation of ground-truth depth and have recently attracted increasing attention. In this work, we propose a self-supervised monocular depth estimation network via binocular geometric correlation learning. Specifically, considering the inter-view geometric correlation, a binocular cue prediction module is presented to generate the auxiliary vision cue for the self-supervised learning of monocular depth estimation. Then, to deal with the occlusion in depth estimation, an occlusion interference attenuated constraint is developed to guide the supervision of the network by inferring the occlusion region and producing paired occlusion masks. Experimental results on two popular benchmark datasets have demonstrated that the proposed network obtains competitive results compared to state-of-the-art self-supervised methods and achieves comparable results to some popular supervised methods.

Inferring the Focal Depths of Small Earthquakes in Southern California Using Physics-Based Waveform Features

ABSTRACT Determining the depths of small crustal earthquakes is challenging in many regions of the world, because most seismic networks are too sparse to resolve trade-offs between depth and origin time with conventional arrival-time methods. Precise and accurate depth estimation is important, because it can help seismologists discriminate between earthquakes and explosions, which is relevant to monitoring nuclear test ban treaties and producing earthquake catalogs that are uncontaminated by mining blasts. Here, we examine the depth sensitivity of several physics-based waveform features for ∼8000 earthquakes in southern California that have well-resolved depths from arrival-time inversion. We focus on small earthquakes (2<ML<4) recorded at local distances (<150 km), for which depth estimation is especially challenging. We find that differential magnitudes (Mw/ML–Mc) are positively correlated with focal depth, implying that coda wave excitation decreases with focal depth. We analyze a simple proxy for relative frequency content, Φ≡log10(M0)+3log10(fc), and find that source spectra are preferentially enriched in high frequencies, or “blue-shifted,” as focal depth increases. We also find that two spectral amplitude ratios Rg 0.5–2 Hz/Sg 0.5–8 Hz and Pg/Sg at 3–8 Hz decrease as focal depth increases. Using multilinear regression with these features as predictor variables, we develop models that can explain 11%–59% of the variance in depths within 10 subregions and 25% of the depth variance across southern California as a whole. We suggest that incorporating these features into a machine learning workflow could help resolve focal depths in regions that are poorly instrumented and lack large databases of well-located events. Some of the waveform features we evaluate in this study have previously been used as source discriminants, and our results imply that their effectiveness in discrimination is partially because explosions generally occur at shallower depths than earthquakes.

Estimating snow depth based on dual polarimetric radar index from Sentinel-1 GRD data: A case study in the Scandinavian Mountains

The sensitivity of synthetic aperture radar (SAR) polarization information to snow depth changes provides new opportunities for regional snow depth retrieval in mountains with thick snow cover. However, interference from soil signals can affect the accurate quantification of snow volume scattering signals. The aim of this study was to develop a dual-polarimetric radar snow depth estimation (DpRSE) methodological framework that utilizes Sentinel-1 data for snow depth retrieval in the Scandinavian Mountains. This framework is based on the dual-polarimetric radar vegetation index (DpRVIc), which enhances the snow volume scattering signal and reduces soil interference by integrating soil purity and a conditioning factor to fully exploit the advantages of polarimetric SAR information. The results revealed a strong correlation between the DpRVIc and snow depth, and the DpRVIc significantly outperformed the other polarimetric radar indices. The multitemporal retrieval results clearly reflect the heterogeneity of the snow depth distribution. This method is applicable to areas without tree cover. Meanwhile, the shrub-filled regions also affect the snow depth retrieval accuracy. Under the same environmental conditions and using the same validation dataset, a comparison between the DpRSE method and the cross-polarization ratio method was conducted. The results indicated that the DpRSE method surpasses similar existing methods in terms of snow depth retrieval accuracy. Specifically, the coefficient of determination (R2) for the DpRSE method reached 0.66, which represents a 26.9 % improvement over that of the cross-polarization ratio method. Moreover, the mean absolute error (MAE) decreased by 20 %. This study offers a novel perspective on SAR snow depth retrieval utilizing polarization information.

Leveraging a realistic synthetic database to learn Shape-from-Shading for estimating the colon depth in colonoscopy images

Colonoscopy is the choice procedure to diagnose, screening, and treat the colon and rectum cancer, from early detection of small precancerous lesions (polyps), to confirmation of malign masses. However, the high variability of the organ appearance and the complex shape of both the colon wall and structures of interest make this exploration difficult. Learned visuospatial and perceptual abilities mitigate technical limitations in clinical practice by proper estimation of the intestinal depth. This work introduces a novel methodology to estimate colon depth maps in single frames from monocular colonoscopy videos. The generated depth map is inferred from the shading variation of the colon wall with respect to the light source, as learned from a realistic synthetic database. Briefly, a classic convolutional neural network architecture is trained from scratch to estimate the depth map, improving sharp depth estimations in haustral folds and polyps by a custom loss function that minimizes the estimation error in edges and curvatures. The network was trained by a custom synthetic colonoscopy database herein constructed and released, composed of 248400 frames (47 videos), with depth annotations at the level of pixels. This collection comprehends 5 subsets of videos with progressively higher levels of visual complexity. Evaluation of the depth estimation with the synthetic database reached a threshold accuracy of 95.65%, and a mean-RMSE of 0.451cm, while a qualitative assessment with a real database showed consistent depth estimations, visually evaluated by the expert gastroenterologist coauthoring this paper. Finally, the method achieved competitive performance with respect to another state-of-the-art method using a public synthetic database and comparable results in a set of images with other five state-of-the-art methods. Additionally, three-dimensional reconstructions demonstrated useful approximations of the gastrointestinal tract geometry. Code for reproducing the reported results and the dataset are available at https://github.com/Cimalab-unal/ColonDepthEstimation.