Depth Estimation Research Articles

A monocular thoracoscopic 3D scene reconstruction framework based on NeRF.

With the increasing use of image-based 3D reconstruction in medical procedures, accurate scene reconstruction plays a crucial role in surgical navigation and assisted treatment. However, the monotonous colors, limited image features, and obvious brightness fluctuations of thoracoscopic scenes make the feature point matching process, on which traditional 3D reconstruction methods rely, unstable and unreliable. It brings a great challenge to accurate 3D reconstruction. In this study, a new method for implicit 3D reconstruction of monocular thoracoscopic scenes is proposed. The method combines a pre-trained metric depth estimation model with neural radiation field (NeRF) technique and uses dense SLAM to accurately compute the camera pose. To ensure the accuracy of the depth values and the structural consistency of the reconstructed scene, depth and normal constraints are added to the original color constraints of the NeRF network to achieve high-quality scene reconstruction results. We conducted experiments on the SCARED dataset and the clinical dataset. After comparing with other methods, the depth estimation accuracy and point cloud reconstruction quality of this paper outperform the existing methods. The method in this paper can provide more accurate 3D reconstruction of complex thoracic surgical scenes, which can significantly improve the accuracy and therapeutic efficacy of surgical navigation.

Improving 3D Reconstruction Through RGB-D Sensor Noise Modeling

High-resolution RGB-D sensors are widely used in computer vision, manufacturing, and robotics. The depth maps from these sensors have inherently high measurement uncertainty that includes both systematic and non-systematic noise. These noisy depth estimates degrade the quality of scans, resulting in less accurate 3D reconstruction, making them unsuitable for some high-precision applications. In this paper, we focus on quantifying the uncertainty in the depth maps of high-resolution RGB-D sensors for the purpose of improving 3D reconstruction accuracy. To this end, we estimate the noise model for a recent high-precision RGB-D structured light sensor called Zivid when mounted on a robot arm. Our proposed noise model takes into account the measurement distance and angle between the sensor and the measured surface. We additionally analyze the effect of background light, exposure time, and the number of captures on the quality of the depth maps obtained. Our noise model seamlessly integrates with well-known classical and modern neural rendering-based algorithms, from KinectFusion to Point-SLAM methods using bilinear interpolation as well as 3D analytical functions. We collect a high-resolution RGB-D dataset and apply our noise model to improve tracking and produce higher-resolution 3D models.

Label-free 3D optical angiography via time-frequency domain analysis of focal modulated dynamic blood flow

Accurate three-dimensional (3D) blood flow imaging with high spatiotemporal resolution and significant detection depth is essential for studying vascular structure-related diseases. In this Letter, we introduce a label-free 3D optical angiography technique via time-frequency domain analysis (TFDA) of focal modulated dynamic blood flow. First, a low-magnification telecentric lens is used for sparse axial sampling within a large depth-of-field range to obtain a coarse estimate of vascular depth. Then, based on the frequency-depth characteristics of dynamic blood flow signals, a TFDA-based focusing evaluation function is established in combination with Lambert–Beer's law, achieving a mean absolute percentage error of 2.06%. Finally, validation on a 3-day-old chicken embryo demonstrated a lateral spatial resolution of 2.95 μm and imaging time of 11.5 s for a 4.95 × 4.95 × 0.7 mm3 sample. Our method provides effective blood flow depth localization by assessing focal modulation intensity relative to focal plane and blood flow position, offering promising support for vascular disease research.

Unsupervised monocular depth estimation with omnidirectional camera for 3D reconstruction of grape berries in the wild.

Japanese table grapes are quite expensive because their production is highly labor-intensive. In particular, grape berry pruning is a labor-intensive task performed to produce grapes with desirable characteristics. Because it is considered difficult to master, it is desirable to assist new entrants by using information technology to show the recommended berries to cut. In this research, we aim to build a system that identifies which grape berries should be removed during the pruning process. To realize this, the 3D positions of individual grape berries need to be estimated. Our environmental restriction is that bunches hang from trellises at a height of about 1.6 meters in the grape orchards outside. It is hard to use depth sensors in such circumstances, and using an omnidirectional camera with a wide field of view is desired for the convenience of shooting videos. Obtaining 3D information of grape berries from videos is challenging because they have textureless surfaces, highly symmetric shapes, and crowded arrangements. For these reasons, it is hard to use conventional 3D reconstruction methods, which rely on matching local unique features. To satisfy the practical constraints of this task, we extend a deep learning-based unsupervised monocular depth estimation method to an omnidirectional camera and propose using it. Our experiments demonstrate the effectiveness of the proposed method for estimating the 3D positions of grape berries in the wild.

HPD-Depth: High performance decoding network for self-supervised monocular depth estimation

Self-supervised monocular depth estimation methods have shown promising results by leveraging geometric relationships among image sequences for network supervision. However, existing methods often face challenges such as blurry depth edges, high computational overhead, and information redundancy. This paper analyzes and investigates technologies related to deep feature encoding, decoding, and regression, and proposes a novel depth estimation network termed HPD-Depth, optimized by three strategies: utilizing the Residual Channel Attention Transition (RCAT) module to bridge the semantic gap between encoding and decoding features while highlighting important features; adopting the Sub-pixel Refinement Upsampling (SPRU) module to obtain high-resolution feature maps with detailed features; and introducing the Adaptive Hybrid Convolutional Attention (AHCA) module to address issues of local depth confusion and depth boundary blurriness. HPD-Depth excels at extracting clear scene structures and capturing detailed local information while maintaining an effective balance between accuracy and parameter count. Comprehensive experiments demonstrate that HPD-Depth performs comparably to state-of-the-art algorithms on the KITTI benchmarks and exhibits significant potential when trained with high-resolution data. Compared with the baseline model, the average relative error and squared relative error are reduced by 6.09% and 12.62% in low-resolution experiments, respectively, and by 11.3% and 18.5% in high-resolution experiments, respectively. Moreover, HPD-Depth demonstrates excellent generalization performance on the Make3D dataset.

Seamless augmented reality integration in arthroscopy: a pipeline for articular reconstruction and guidance.

Arthroscopy is a minimally invasive surgical procedure used to diagnose and treat joint problems. The clinical workflow of arthroscopy typically involves inserting an arthroscope into the joint through a small incision, during which surgeons navigate and operate largely by relying on their visual assessment through the arthroscope. However, the arthroscope's restricted field of view and lack of depth perception pose challenges in navigating complex articular structures and achieving surgical precision during procedures. Aiming at enhancing intraoperative awareness, a robust pipeline that incorporates simultaneous localization and mapping, depth estimation, and 3D Gaussian splatting (3D GS) is presented to realistically reconstruct intra-articular structures solely based on monocular arthroscope video. Extending 3D reconstruction to augmented reality (AR) applications, the solution offers AR assistance for articular notch measurement and annotation anchoring in a human-in-the-loop manner. Compared to traditional structure-from-motion and neural radiance field-based methods, the pipeline achieves dense 3D reconstruction and competitive rendering fidelity with explicit 3D representation in 7 min on average. When evaluated on four phantom datasets, our method achieves root-mean-square-error reconstruction error, peak signal-to-noise ratio and structure similarity index measure on average. Because the pipeline enables AR reconstruction and guidance directly from monocular arthroscopy without any additional data and/or hardware, the solution may hold the potential for enhancing intraoperative awareness and facilitating surgical precision in arthroscopy. The AR measurement tool achieves accuracy within and the AR annotation tool achieves a mIoU of 0.721.

Efficient Light Field Transmission via Enhanced Resampling Reconstruction and Angular Attention Estimation

Light field (LF) imaging technology has significantly enhanced immersive virtual reality experiences and advanced computer vision tasks like depth estimation and 3D reconstruction. However, the massive data volume of light field images (LFIs) presents a transmission challenge. To address this issue, we propose a novel intelligent LF transmission approach integrating enhanced resampling reconstruction and angular attention estimation. The proposed enhanced resampling reconstruction method reduces spatial redundancy by downsampling before transmission and upsampling afterward, aided by a pre-calculated residual map to minimizing quality loss. To reduce angular redundancy while preserving perceptual quality, we designed a LF angular attention estimation network employing specifically designed angular attention kernels to guide differential transmission within the angular domain. To train this network, we construct the first LF eye-tracking dataset, featuring numerous of samples, diverse realistic scenes, and a wealth of data elements, particularly real user angular attention data, establishing it a key benchmark in LF transmission. Extensive experiments demonstrate that our transmission strategy decreases transmission time by 97.3% with only a 3.1% perceptual quality loss. Subjective experiments further confirm the superior performance of our method. An implementation of this paper and the constructed dataset are available at https://github.com/VincentQQu/LF-Transmission.

LungDepth: Self-Supervised Multi-Frame Monocular Depth Estimation for Bronchoscopy.

Bronchoscopy is an essential measure for conducting lung biopsies in clinical practice. It is crucial for advancing the intelligence of bronchoscopy to acquire depth information from bronchoscopic image sequences. A self-supervised multi-frame monocular depth estimation approach for bronchoscopy is constructed. Networks are trained by minimising the photometric reprojection error between the target frame and the reconstructed target frame. The adaptive dual attention module and the details emphasis module are introduced to better capture the edge contour and internal details. In addition, the approach is evaluated on a self-made dataset and compared against other established methods. Experimental results demonstrate that the proposed method outperforms other self-supervised monocular depth estimation approaches in both quantitative measurement and qualitative analysis. Our monocular depth estimation approach for bronchoscopy achieves superior performance in terms of error and accuracy, and passes physical model validations, which can facilitate further research into intelligent bronchoscopic procedures.

A tightly-coupled dense monocular Visual-Inertial Odometry system with lightweight depth estimation network

A tightly-coupled dense monocular Visual-Inertial Odometry system with lightweight depth estimation network

Monocular depth estimation based on deep learning for intraoperative guidance using surface-enhanced Raman scattering imaging

Imaging of surface-enhanced Raman scattering (SERS) nanoparticles (NPs) has been intensively studied for cancer detection due to its high sensitivity, unconstrained low signal-to-noise ratios, and multiplexing detection capability. Furthermore, conjugating SERS NPs with various biomarkers is straightforward, resulting in numerous successful studies on cancer detection and diagnosis. However, Raman spectroscopy only provides spectral data from an imaging area without co-registered anatomic context. This is not practical and suitable for clinical applications. Here, we propose a custom-made Raman spectrometer with computer-vision-based positional tracking and monocular depth estimation using deep learning (DL) for the visualization of 2D and 3D SERS NPs imaging, respectively. In addition, the SERS NPs used in this study (hyaluronic acid-conjugated SERS NPs) showed clear tumor targeting capabilities (target CD44 typically overexpressed in tumors) by an ex vivo experiment and immunohistochemistry. The combination of Raman spectroscopy, image processing, and SERS molecular imaging, therefore, offers a robust and feasible potential for clinical applications.

A Light Field Depth Estimation Algorithm Considering Blur Features and Prior Knowledge of Planar Geometric Structures

A Light Field Depth Estimation Algorithm Considering Blur Features and Prior Knowledge of Planar Geometric Structures

Using applications of machine learning models for predicting and analyzing scour depth at the submerged weir

ABSTRACT Weirs are designed to stabilize rivers, grade control, and raise upstream water levels. The failure of these structures is primarily due to local scour at the structural site. Consequently, an accurate estimate of the likely scour depth at the structure is critical for weir design safety and economy. This study proposes machine learning models for scour depth prediction at submerged weirs by introducing advanced gradient boosting algorithms, namely gradient boosting (GB), categorical boosting (CatBoost), adaptive boosting (AdaBoost), and extreme gradient boosting (XGBoost). A database consisting of 308 cases was collected for model calibration and evaluation. The results demonstrate that the GB algorithm is very accurate, with coefficients of determination of 0.99610 and 0.96222 for the training and testing datasets, respectively. The GB model outperforms other developed models, such as support vector regression, decision tree, and ridge models, in the literature. A sensitivity analysis study has determined that the morphological jump parameter is the most significant factor, whereas the normal flow depth on the equilibrium bed slope parameter is the least significant factor in predicting the ds under the submerged weir.

MonoAMP: Adaptive Multi-Order Perceptual Aggregation for Monocular 3D Vehicle Detection

Monocular 3D object detection is rapidly emerging as a key research direction in autonomous driving, owing to its resource efficiency and ease of implementation. However, existing methods face certain limitations in cross-dimensional feature attention mechanisms and multi-order contextual information modeling, which constrain their detection performance in complex scenes. Thus, we propose MonoAMP, an adaptive multi-order perceptual aggregation algorithm for monocular 3D object detection. We first introduce triplet attention to enhance the interaction of cross-dimensional feature attention. Second, we design an adaptive multi-order perceptual aggregation module. It dynamically captures multi-order contextual information and employs an adaptive aggregation strategy to enhance target perception. Finally, we propose an uncertainty-guided adaptive depth ensemble strategy, which models the uncertainty distribution in depth estimation and effectively fuses multiple depth predictions. Experiments demonstrate that MonoAMP significantly enhances performance on the KITTI dataset at the moderate difficulty level, achieving 16.80% AP3D and 24.47% APBEV. Additionally, the ablation study shows a 3.78% improvement in object detection accuracy over the baseline method. Compared to other advanced methods, MonoAMP demonstrates superior detection capabilities, especially in complex scenarios.

Bulk Motions in the Black Hole Jet Sheath as a Candidate for the Comptonizing Corona

Abstract Using two-dimensional general relativistic resistive magnetohydrodynamic simulations, we investigate the properties of the sheath separating the black hole jet from the surrounding medium. We find that the electromagnetic power flowing through the jet sheath is comparable to the overall accretion power of the black hole. The sheath is an important site of energy dissipation as revealed by the copious appearance of reconnection layers and plasmoid chains. About 20% of the sheath power is dissipated between 2 and 10 gravitational radii. The plasma in the dissipative sheath moves along a nearly paraboloidal surface with transrelativistic bulk motions dominated by the radial component, whose dimensionless 4-velocity is ∼1.2 ± 0.5. In the frame moving with the mean (radially dependent) velocity, the distribution of stochastic bulk motions resembles a Maxwellian with an “effective bulk temperature” of ∼100 keV. Scaling the global simulation to Cygnus X-1 parameters gives a rough estimate of the Thomson optical depth across the jet sheath, ∼0.01–0.1, and it may increase in future magnetohydrodynamic simulations with self-consistent radiative losses. These properties suggest that the dissipative jet sheath may be a viable “coronal” region, capable of upscattering seed soft photons into a hard, nonthermal tail, as seen during the hard states of X-ray binaries and active galactic nuclei.

Applying auxiliary supervised depth-assisted transformer and cross modal attention fusion in monocular 3D object detection.

Monocular 3D object detection is the most widely applied and challenging solution for autonomous driving, due to 2D images lacking 3D information. Existing methods are limited by inaccurate depth estimations by inequivalent supervised targets. The use of both depth and visual features also faces problems of heterogeneous fusion. In this article, we propose Depth Detection Transformer (Depth-DETR), applying auxiliary supervised depth-assisted transformer and cross modal attention fusion in monocular 3D object detection. Depth-DETR introduces two additional depth encoders besides the visual encoder. Two depth encoders are supervised by ground truth depth and bounding box respectively, working independently to complement each other's limitations and predicting more accurate target distances. Furthermore, Depth-DETR employs cross modal attention mechanisms to effectively fuse three different features. A parallel structure of two cross modal transformer is applied to fuse two depth features with visual features. Avoiding early fusion between two depth features enhances the final fused feature for better feature representations. Through multiple experimental validations, the Depth-DETR model has achieved highly competitive results in the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset, with an AP score of 17.49, representing its outstanding performance in 3D object detection.

Brief communication: Monitoring snow depth using small, cheap, and easy-to-deploy snow–ground interface temperature sensors

Abstract. Temporally continuous snow depth estimates are vital for understanding changing snow patterns and impacts on permafrost in the Arctic. We trained a random forest machine learning model to predict snow depth from variability in snow–ground interface temperature. The model performed well on Alaska's Seward Peninsula where it was trained and at Arctic evaluation sites (RMSE ≤ 0.15 m). It performed poorly at temperate sites with deeper snowpacks, partially due to training data limitations. Small temperature sensors are cheap and easy to deploy, so this technique enables spatially distributed and temporally continuous snowpack monitoring at high latitudes to an extent previously infeasible.

Speckle-illumination spatial frequency domain imaging with a stereo laparoscope for profile-corrected optical property mapping.

Laparoscopic surgery presents challenges in localizing oncological margins due to poor contrast between healthy and malignant tissues. Optical properties can uniquely identify various tissue types and disease states with high sensitivity and specificity, making it a promising tool for surgical guidance. Although spatial frequency domain imaging (SFDI) effectively measures quantitative optical properties, its deployment in laparoscopy is challenging due to the constrained imaging environment. Thus, there is a need for compact structured illumination techniques to enable accurate, quantitative endogenous contrast in minimally invasive surgery. We introduce a compact, two-camera laparoscope that incorporates both active stereo depth estimation and speckle-illumination SFDI (si-SFDI) to map profile-corrected, pixel-level absorption ( ), and reduced scattering ( ) optical properties in images of tissues with complex geometries. We used a multimode fiber-coupled 639-nm laser illumination to generate high-contrast speckle patterns on the object. These patterns were imaged through a modified commercial stereo laparoscope for optical property estimation via si-SFDI. Compared with the original si-SFDI work, which required images of randomized speckle patterns for accurate optical property estimations, our approach approximates the DC response using a laser speckle reducer (LSR) and consequently requires only two images. In addition, we demonstrate 3D profilometry using active stereo from low-coherence RGB laser flood illumination. Sample topography was then used to correct for measured intensity variations caused by object height and surface angle differences with respect to a calibration phantom. The low-contrast RGB speckle pattern was blurred using an LSR to approximate incoherent white light illumination. We validated profile-corrected si-SFDI against conventional SFDI in phantoms with simple and complex geometries, as well as in a human finger in vivo time-series constriction study. Laparoscopic si-SFDI optical property measurements agreed with conventional SFDI measurements when measuring flat tissue phantoms, exhibiting an error of 6.4% for absorption and 5.8% for reduced scattering. Profile-correction improved the accuracy for measurements of phantoms with complex geometries, particularly for absorption, where it reduced the error by 23.7%. An in vivo finger constriction study further validated laparoscopic si-SFDI, demonstrating an error of 8.2% for absorption and 5.8% for reduced scattering compared with conventional SFDI. Moreover, the observed trends in optical properties due to physiological changes were consistent with previous studies. Our stereo-laparoscopic implementation of si-SFDI provides a simple method to obtain accurate optical property maps through a laparoscope for flat and complex geometries. This has the potential to provide quantitative endogenous contrast for minimally invasive surgical guidance.

Advancing terrestrial snow depth monitoring with machine learning and L-band InSAR data: a case study using NASA’s SnowEx 2017 data

Current terrestrial snow depth mapping from space faces challenges in spatial coverage, revisit frequency, and cost. Airborne lidar, although precise, incurs high costs and has limited geographical coverage, thereby necessitating the exploration of alternative, cost-effective methodologies for snow depth estimation. The forthcoming NASA-ISRO Synthetic Aperture Radar (NISAR) mission, with its 12-day global revisit cycle and 1.25 GHz L-band frequency, introduces a promising avenue for cost-effective, large-scale snow depth and snow water equivalent (SWE) estimation using L-band Interferometric SAR (InSAR) capabilities. This study demonstrates InSAR’s potential for snow depth estimation via machine learning. Using 3 m resolution L-band InSAR products over Grand Mesa, Colorado, we compared the performance of three machine learning approaches (XGBoost, ExtraTrees, and Neural Networks) across open, vegetated, and the combined (open + vegetated) datasets using Root Mean Square Error (RMSE), Mean Bias Error (MBE), and R2 metrics. XGBoost emerged as the superior model, with RMSE values of 9.85 cm, 10.46 cm, and 9.88 cm for open, vegetated, and combined regions, respectively. Validation against in situ snow depth measurements resulted in an RMSE of approximately 16 cm, similar to in situ validation of the airborne lidar. Our findings indicate that L-band InSAR, with its ability to penetrate clouds and cover extensive areas, coupled with machine learning, holds promise for enhancing snow depth estimation. This approach, especially with the upcoming NISAR launch, may enable high-resolution (∼10 m) snow depth mapping over extensive areas, provided suitable training data are available, offering a cost-effective approach for snow monitoring. The code and data used in this work are available at https://github.com/cryogars/uavsar-lidar-ml-project.

Identification of clinical needs for the improvement of ai-assisted colonoscopy cad systems

Abstract Introduction Advances in artificial intelligence (AI)-assisted colonoscopy have enabled the detection and classification of polyps using computer-aided diagnosis (CAD) systems. However, there is still room for improvement, particularly in classification (CADx), and other clinical needs remain unaddressed. Methods To identify these needs, several meetings were held between expert groups with both clinical and technical backgrounds to recognise clinically relevant needs that are technically feasible. Results Some of the identified needs include: (i) identification of additional pathologies, such as diverticula or cancer invasion depth estimation; (ii) identification of the current location within the colon (including cecum identification); (iii) assessment of colon cleanliness and preparation quality (e.g., using the Boston Bowel Preparation Scale), both locally and globally; (iv) development of an alert system to guide explorations and avoid unexplored areas; and (v) automatic generation of colonoscopy reports, including a count of polyps found during the examination. Conclusion Several clinically feasible needs have been identified for the improvement of current AI-assisted CAD systems, which could further enhance the quality of examinations and optimise clinical workflows.

Optimization of Underwater Images Based on Gray World Algorithm and Jaffe-McGlamery Models

In this paper, a comprehensive scheme for underwater image processing is proposed based on the grayscale world algorithm and the Jaffe-McGlamery model. Firstly, a color bias detection based on grayscale world theory, a low light detection based on HSV color space, and a fuzzy detection method based on frequency domain and Laplace operator are designed to classify different types of image degradation. Subsequently, the corresponding scene degradation models are constructed for different degradation types through the simplified Jaffe-McGlamery model, and the image features under different water conditions are analyzed. Next, an improved gray world algorithm is used to eliminate color bias, the limiting contrast adaptive histogram equalization (CLAHE) technique is utilized to improve the image quality under low-light conditions, and the dark-channel a priori algorithm is optimized by the depth estimation and parameter adaptation modules to remove blurring. The proposed method in this study significantly improves the image quality in different scenarios and provides a new idea for underwater image processing.