Depth Estimation Techniques Research Articles

Long-term reprojection loss for self-supervised monocular depth estimation in endoscopic surgery

Aim: Depth information plays a key role in enhanced perception and interaction in image-guided surgery. However, it is difficult to obtain depth information with monocular endoscopic surgery due to a lack of reliable cues for perceiving depth. Although there are reprojection loss-based self-supervised learning techniques to estimate depth and pose, the temporal information from the adjacent frames is not efficiently utilized to handle occlusion in surgery. Methods: We design long-term reprojection loss (LT-RL) self-supervised monocular depth estimation techniques by integrating longer temporal sequences into reprojection to learn better perception and to address occlusion artifacts in image-guided laparoscopic and robotic surgery. For this purpose, we exploit four temporally adjacent source frames before and after the target frame, where conventional reprojection loss uses two adjacent frames. The pixels that are visible in the target frame but occluded in the immediate two adjacent frames will produce the inaccurate depth but a higher chance to appear in the four adjacent frames during the calculation of minimum reprojection loss. Results: We validate LT-RL on the benchmark surgical datasets of Stereo correspondence and reconstruction of endoscopic data (SCARED) and Hamlyn to compare the performance with other state-of-the-art depth estimation methods. The experimental results show that our proposed technique yields 2%-4% better root-mean-squared error (RMSE) over the baselines of vanilla reprojection loss. Conclusion: Our LT-RL self-supervised depth and pose estimation technique is a simple yet effective method to tackle occlusion artifacts in monocular surgical video. It does not add any training parameters, making it flexible for integration with any network architecture and improving the performance significantly.

Monocular Depth Estimation via Self-Supervised Self-Distillation.

Self-supervised monocular depth estimation can exhibit excellent performance in static environments due to the multi-view consistency assumption during the training process. However, it is hard to maintain depth consistency in dynamic scenes when considering the occlusion problem caused by moving objects. For this reason, we propose a method of self-supervised self-distillation for monocular depth estimation (SS-MDE) in dynamic scenes, where a deep network with a multi-scale decoder and a lightweight pose network are designed to predict depth in a self-supervised manner via the disparity, motion information, and the association between two adjacent frames in the image sequence. Meanwhile, in order to improve the depth estimation accuracy of static areas, the pseudo-depth images generated by the LeReS network are used to provide the pseudo-supervision information, enhancing the effect of depth refinement in static areas. Furthermore, a forgetting factor is leveraged to alleviate the dependency on the pseudo-supervision. In addition, a teacher model is introduced to generate depth prior information, and a multi-view mask filter module is designed to implement feature extraction and noise filtering. This can enable the student model to better learn the deep structure of dynamic scenes, enhancing the generalization and robustness of the entire model in a self-distillation manner. Finally, on four public data datasets, the performance of the proposed SS-MDE method outperformed several state-of-the-art monocular depth estimation techniques, achieving an accuracy (δ1) of 89% while minimizing the error (AbsRel) by 0.102 in NYU-Depth V2 and achieving an accuracy (δ1) of 87% while minimizing the error (AbsRel) by 0.111 in KITTI.

Self-supervised monocular depth estimation on water scenes via specular reflection prior

Monocular depth estimation from a single image is an ill-posed problem for computer vision due to insufficient reliable cues as the prior knowledge. Besides the inter-frame supervision, namely stereo and adjacent frames, extensive prior information is available in the same frame. Reflections from specular surfaces, informative intra-frame priors, enable us to reformulate the ill-posed depth estimation task as a multi-view synthesis. This paper proposes the first self-supervision for deep-learning depth estimation on water scenes via intra-frame priors, known as reflection supervision and geometrical constraints. In the first stage, a water segmentation network is performed to separate the reflection components from the entire image. Next, we construct a self-supervised framework to predict the target appearance from reflections, perceived as other perspectives. The photometric re-projection error, incorporating SmoothL1 and a novel photometric adaptive SSIM, is formulated to optimize pose and depth estimation by aligning the transformed virtual depths and source ones. As a supplement, the water surface is determined from real and virtual camera positions, which complement the depth of the water area. Furthermore, to alleviate these laborious ground truth annotations, we introduce a large-scale water reflection scene (WRS) dataset rendered from Unreal Engine 4. Extensive experiments on the WRS dataset prove the feasibility of the proposed method compared to state-of-the-art depth estimation techniques.

A Survey of Monocular Depth Estimation based on Deep Learning

Depth information is very important for machines to perceive the environment and estimate their own state. Significant advances in robotics engineering and self-driving cars in recent decades have increased the demand for accurate depth measurements. Traditional depth estimation methods include motion structure and stereo vision matching, but these are based on the feature correspondence of multiple viewpoints, and at the same time, the predicted depth map is sparse. Depth estimation is a traditional task in computer vision that can be properly predicted by applying a variety of procedures, whereas inferring depth information from a single image is an ill-posed problem. The main objective of this paper is to provide a brief overview of the development of monocular depth estimation techniques based on deep learning. This article attempts to give an overview of supervised, unsupervised, and datasets and evaluation metrics. We conclude with a brief analysis of future developments.

PS 2 F: Polarized Spiral Point Spread Function for Single-Shot 3D Sensing.

We propose a compact snapshot monocular depth estimation technique that relies on an engineered point spread function (PSF). Traditional approaches used in microscopic super-resolution imaging such as the Double-Helix PSF (DHPSF) are ill-suited for scenes that are more complex than a sparse set of point light sources. We show, using the Cramér-Rao lower bound, that separating the two lobes of the DHPSF and thereby capturing two separate images leads to a dramatic increase in depth accuracy. A special property of the phase mask used for generating the DHPSF is that a separation of the phase mask into two halves leads to a spatial separation of the two lobes. We leverage this property to build a compact polarization-based optical setup, where we place two orthogonal linear polarizers on each half of the DHPSF phase mask and then capture the resulting image with a polarization-sensitive camera. Results from simulations and a lab prototype demonstrate that our technique achieves up to 50% lower depth error compared to state-of-the-art designs including the DHPSF and the Tetrapod PSF, with little to no loss in spatial resolution.

Safe Distance Monitoring of Live Equipment Based Upon Instance Segmentation and Pseudo-LiDAR

Electrocution accidents caused by operation and maintenance personnel and high-voltage live equipment frequently occur in substations. Although many cameras have been installed for the surveillance of critical energized equipment, they cannot complete the monitoring of safe distance. In addition, the current research based on vision or LiDAR suffers from computationally intensive, lack of real-time or expensive equipment. In order to meet the low cost, easy maintenance and real-time requirements of substation safe distance monitoring, a monocular vision method based on 2D-3D fusion is proposed in this study. Specifically, instance segmentation, depth estimation, depth reconstruction, and back projection transformation are used to predict the 3D distance of objects in 2D images. The work is mainly implemented in two aspects: a) To obtain the high-quality masks of Power Transformers and Person, SOLOv2 is optimized on three aspects: feature extraction, feature fusion and ability to cope with object geometric transformation. b) Pseudo-LiDAR output from RGB images via depth estimation and back projection techniques. Subsequently, combined with the point cloud of the object and the actual-to-virtual conversion ratio, the actual distance between Power Transformer and Person can be calculated. Experimental results show that the maximum error of ranging is within 9%, with an average error rate of 5.34% on the Power Transformer-Person dataset. It can be seen that the proposed method has achieved good ranging effects, and can realize the automatic measurement of the safe distance of Power Transformer.

Probabilistic multi-modal depth estimation based on camera–LiDAR sensor fusion

Multi-modal depth estimation is one of the key challenges for endowing autonomous machines with robust robotic perception capabilities. There have been outstanding advances in the development of uni-modal depth estimation techniques based on either monocular cameras, because of their rich resolution, or LiDAR sensors, due to the precise geometric data they provide. However, each of these suffers from some inherent drawbacks, such as high sensitivity to changes in illumination conditions in the case of cameras and limited resolution for the LiDARs. Sensor fusion can be used to combine the merits and compensate for the downsides of these two kinds of sensors. Nevertheless, current fusion methods work at a high level. They process the sensor data streams independently and combine the high-level estimates obtained for each sensor. In this paper, we tackle the problem at a low level, fusing the raw sensor streams, thus obtaining depth estimates which are both dense and precise, and can be used as a unified multi-modal data source for higher-level estimation problems. This work proposes a conditional random field model with multiple geometry and appearance potentials. It seamlessly represents the problem of estimating dense depth maps from camera and LiDAR data. The model can be optimized efficiently using the conjugate gradient squared algorithm. The proposed method was evaluated and compared with the state of the art using the commonly used KITTI benchmark dataset.

Robust 3D breast reconstruction based on monocular images and artificial intelligence for robotic guided oncological interventions.

Breast cancer is a global public health concern. For women with suspicious breast lesions, the current diagnosis requires a biopsy, which is usually guided by ultrasound (US). However, this process is challenging due to the low quality of the US image and the complexity of dealing with the US probe and the surgical needle simultaneously, making it largely reliant on the surgeon's expertise. Some previous works employing collaborative robots emerged to improve the precision of biopsy interventions, providing an easier, safer, and more ergonomic procedure. However, for these equipment to be able to navigate around the breast autonomously, 3D breast reconstruction needs to be available. The accuracy of these systems still needs to improve, with the 3D reconstruction of the breast being one of the biggest focuses of errors. The main objective of this work is to develop a method to obtain a robust 3D reconstruction of the patient's breast, based on RGB monocular images, which later can be used to compute the robot's trajectories for the biopsy. To this end, depth estimation techniques will be developed, based on a deep learning architecture constituted by a CNN, LSTM, and MLP, to generate depth maps capable of being converted into point clouds. After merging several from multiple points of view, it is possible to generate a real-time reconstruction of the breast as a mesh. The development and validation of our method was performed using a previously described synthetic dataset. Hence, this procedure takes RGB images and the cameras' position and outputs the breasts' meshes. It has a mean error of 3.9 mm and a standard deviation of 1.2 mm. The final results attest to the ability of this methodology to predict the breast's shape and size using monocular images.Clinical Relevance- This work proposes a method based on artificial intelligence and monocular RGB images to obtain the breast's volume during robotic guided breast biopsies, improving their execution and safety.

228 × 304 200-mW lidar based on a single-point global-depth d-ToF sensor and RGB-guided super-resolution neural network.

The cutting-edge imaging system exhibits low output resolution and high power consumption, presenting challenges for the RGB-D fusion algorithm. In practical scenarios, aligning the depth map resolution with the RGB image sensor is a crucial requirement. In this Letter, the software and hardware co-design is considered to implement a lidar system based on the monocular RGB 3D imaging algorithm. A 6.4 × 6.4-mm2 deep-learning accelerator (DLA) system-on-chip (SoC) manufactured in a 40-nm CMOS is incorporated with a 3.6-mm2 TX-RX integrated chip fabricated in a 180-nm CMOS to employ the customized single-pixel imaging neural network. In comparison to the RGB-only monocular depth estimation technique, the root mean square error is reduced from 0.48 m to 0.3 m on the evaluated dataset, and the output depth map resolution matches the RGB input.

Improving the planarity and sharpness of monocularly estimated depth images using the Phong reflection model

Depth estimation based on a single RGB image (a.k.a. monocular depth estimation) is a challenging task, with applications in robotics, autonomous vehicles, and other areas. With the advance of deep learning, several monocular depth estimation approaches have emerged with remarkable results. However, it is still possible to observe deficiencies in the depth maps generated by current techniques. We present a deep neural network that takes the result produced by some existing monocular depth estimation approach and enhances it by adding the details that the depth map requires to be sharp. We train the proposed Depth Enhancer Neural Network (DENN) using a new loss function that compares the input color image of the scene to the color image produced by rendering it using the Phong reflection model. Our experiments show a clear visual improvement in the sharpness of the depth image produced by the DENN, leading to edge enhancement and regularity of planar surfaces without compromising non-planar objects. On average, the standard metrics show a 5% reduction in error when considering the quantitative results. This reduction is dependent on the initial monocular depth estimation technique used by DENN to train the model.

A lumen-adapted navigation scheme with spatial awareness from monocular vision for autonomous robotic endoscopy

Lumen is a common anatomical condition in endoscopic therapy or diagnostics. The navigation of autonomous robotic endoscopy usually involves vision techniques such as detection of lumen center or anatomically specific contour features. However, these methods may fail to achieve smooth interventions and result in large conveying force without spatial awareness of the tissue state. In this paper, a novel navigation pipeline based on a spatial-aware monocular vision is proposed. The spatial awareness pipeline starts with a data-driven depth estimation technique for reconstructing real-time approximate tissue surface. The spatial shape of the lumen described by the skeleton is then extracted using digital topology. We modify the skeleton to get a smooth pathway, and design an adaptive autonomous control strategy using geometric information from the pathway. We conduct experiments on a colon phantom and ex vivo pig intestines. We test turning performance of several bending segments with different angles in phantom, as well as the overall performance of a long-range intervention task in the phantom and pig intestines. The results show our navigation scheme achieve smoother intervention with lower conveying force. The proposed navigation method with spatial awareness can effectively improve the fluency of autonomous robotic endoscopy.

Deep Monocular Depth Estimation Based on Content and Contextual Features.

Recently, significant progress has been achieved in developing deep learning-based approaches for estimating depth maps from monocular images. However, many existing methods rely on content and structure information extracted from RGB photographs, which often results in inaccurate depth estimation, particularly for regions with low texture or occlusions. To overcome these limitations, we propose a novel method that exploits contextual semantic information to predict precise depth maps from monocular images. Our approach leverages a deep autoencoder network incorporating high-quality semantic features from the state-of-the-art HRNet-v2 semantic segmentation model. By feeding the autoencoder network with these features, our method can effectively preserve the discontinuities of the depth images and enhance monocular depth estimation. Specifically, we exploit the semantic features related to the localization and boundaries of the objects in the image to improve the accuracy and robustness of the depth estimation. To validate the effectiveness of our approach, we tested our model on two publicly available datasets, NYU Depth v2 and SUN RGB-D. Our method outperformed several state-of-the-art monocular depth estimation techniques, achieving an accuracy of 85%, while minimizing the error Rel by 0.12, RMS by 0.523, and log10 by 0.0527. Our approach also demonstrated exceptional performance in preserving object boundaries and faithfully detecting small object structures in the scene.

GFI-Net: Global Feature Interaction Network for Monocular Depth Estimation

Monocular depth estimation techniques are used to recover the distance from the target to the camera plane in an image scene. However, there are still several problems, such as insufficient estimation accuracy, the inaccurate localization of details, and depth discontinuity in planes parallel to the camera plane. To solve these problems, we propose the Global Feature Interaction Network (GFI-Net), which aims to utilize geometric features, such as object locations and vanishing points, on a global scale. In order to capture the interactive information of the width, height, and channel of the feature graph and expand the global information in the network, we designed a global interactive attention mechanism. The global interactive attention mechanism reduces the loss of pixel information and improves the performance of depth estimation. Furthermore, the encoder uses the Transformer to reduce coding losses and improve the accuracy of depth estimation. Finally, a local-global feature fusion module is designed to improve the depth map's representation of detailed areas. The experimental results on the NYU-Depth-v2 dataset and the KITTI dataset showed that our model achieved state-of-the-art performance with full detail recovery and depth continuation on the same plane.

TP-GAN: Simple Adversarial Network With Additional Player for Dense Depth Image Estimation

We present a simple yet robust monocular depth estimation technique by synthesizing a depth map image from a single RGB input image using the advantage of generative adversarial networks (GAN). We employ an additional sub-model termed refiner to extract local depth features, then combine it with the global scene information from the generator to improve the GAN’s performance compared to the standard GAN architectural scheme. Notably, the generator is the first player to learn to synthesize depth images. The second player, the discriminator, classifies the generated depth. In the meantime, the third player, the refiner, enhances the final reconstructed depth. Complementing the GAN model, we apply a conditional generative network (cGAN) to lead the generator in mapping the input image to the respective depth representation. We further incorporate a structured similarity (SSIM) as our loss function for the generator and refiner in GAN training. Through extensive experiment validation, we confirmed the performance of our strategy on the publicly indoor NYU Depth v2 and KITTI outdoor data. Experiment results on the NYU depth v2 dataset show that our proposed approach achieves the best performance by 96.0% on threshold accuracy ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\delta &lt; 1.25^{2}$ </tex-math></inline-formula> ) and the second-best accuracy on all thresholds on the KITTI dataset. We discovered that our proposed method compares favorably to numerous existing monocular depth estimation strategies and demonstrates a considerable improvement in the accuracy of image depth estimation despite its simple network architecture.

Monocular Depth Estimation of Old Photos via Collaboration of Monocular and Stereo Networks

Old photos that were captured about a century ago have archaeological and historical significance. Many of the old photos have been successfully digitized, but most of them suffer from severe and complicated distortion. Thus, prior studies have focused on image restoration tasks such as denoising, inpainting, and colorization. In this paper, we pay attention to the depth estimation of old photos, enabling a more enjoyable appreciation of them and helping better understand past human life, activities, and environments. Because most old photos are available as single-view images, monocular depth estimation techniques can be considered a solution. However, most high-performance techniques are based on supervised learning, which requires ground-truth depth maps. Because this kind of supervised learning is not feasible for old photos, in this paper, we present a learning framework that finetunes a pretrained monocular depth estimation network for each old photo. Specifically, the pretrained monocular depth estimation network predicts stereo depth maps for stereo image rendering. Then, the pretrained stereo network predicts depth estimates from the rendered stereo image pair. By extracting reliable depth estimates and using them for supervision of the monocular network, the monocular network can be gradually learned to produce a high-quality depth map of the given old photo. From the qualitative and quantitative performance evaluations on old photos, we demonstrate the effectiveness of the proposed method.

Depth Estimation using DNN Architecture and Vision-Based Transformers

Depth Estimation is the process of estimating the depth of objects with a 2D image as an input. The importance of Depth Estimation is that it provides some critical information about the 3D structure of a scene given a sequence of 2D images. This is helpful in various applications like robotics, virtual reality, autonomous driving, medical imaging, and so on and so forth. Our main objective here is to make the environment more perceivable by autonomous vehicles. Unlike the traditional approaches which try to extract depth from the input image, we apply instance segmentation to the image and then use the segmented image as an input to the depth map generator. In this paper, we use a fully connected neural network with dense prediction transformers. As mentioned above, the image after instance segmentation is given as an input for the transformers. The Transformer is the backbone for producing high-resolution images. Our experimentation results have shown many improvements related to the details and sharpness of the image as compared to the traditional depth estimation techniques. The architecture is applied and tested on the KITTI data set.

Interpretation of magnetic anomaly profiles using a decomposition in Hilbert transform pairs

ABSTRACT. We propose a simple transformation to aid the interpretation of magnetic anomalies generated by linear structures. The profile of such anomalies perpendicular to the strike can be decomposed into two signals, one symmetric and the other antisymmetric concerning the center of the source. The symmetric component serves as input data to various depth estimation techniques that often assume the anomaly is reduced to the pole. We use the fact that these components form a Hilbert transform pair to transform a skewed anomaly profile into a symmetric one. Unlike in previous works that rely on the decomposition into even and odd functions, the profile does not need to be shifted to the source's center of symmetry or limited to one isolated anomaly. Multiple effective magnetization directions presented by different dikes are modeled by a function representing the different local effective dip angles. We validate the method with synthetic data and ground magnetic survey data from a dike swarm at Ponta Grossa Arch, southern Brazil. We also illustrate the usefulness of reconstructed anomalies for depth estimation methods. The results also show that the method can handle interfering sources with distinct effective magnetization directions.

Computer-based airway stenosis quantification from bronchoscopic images: preliminary results from a feasibility trial.

Airway Stenosis (AS) is a condition of airway narrowing in the expiration phase. Bronchoscopy is a minimally invasive pulmonary procedure used to diagnose and/or treat AS. The AS quantification in a form of the Stenosis Index (SI), whether subjective or digital, is necessary for the physician to decide on the most appropriate form of treatment. The literature reports that the subjective SI estimation is inaccurate. In this paper, we propose an approach to quantify the SI defining the level of airway narrowing, using depth estimation from a bronchoscopic image. In this approach we combined a generative depth estimation technique combined with depth thresholding to provide Computer-based AS quantification. We performed an interim clinical analysis by comparing AS quantification performance of three expert bronchoscopists against the proposed Computer-based method on seven patient datasets. The Mean Absolute Error of the subjective Human-based and the proposed Computer-based SI estimation was [Formula: see text] [%] and [Formula: see text] [%], respectively. The correlation coefficients between the CT measurements were used as the gold standard, and the Human-based and Computer-based SI estimation were [Formula: see text] and 0.46, respectively. We presented a new computer method to quantify the severity of AS in bronchoscopy using depth estimation and compared the performance of the method against a human-based approach. The obtained results suggest that the proposed Computer-based AS quantification is a feasible tool that has the potential to provide significant assistance to physicians in bronchoscopy.

Shape Reconstruction in Computer Vision

The fundamental job of shape reconstruction in computer vision is crucial for many different applications, including robotics, medical imaging, and autonomous systems. The state-of-the-art methods, difficulties, and potential future directions in the field of shape reconstruction are succinctly outlined in this abstract. Inferring the three-dimensional (3D) geometry of objects or scenes from two-dimensional (2D) photographs or point clouds is the process of shape reconstruction. The conventional approaches relied on depth sensors, structure from motion, and stereo vision. Convolutional neural networks (CNNs) and generative models, in particular, are deep learning techniques that have been used recently to achieve impressive results in shape reconstruction. This abstract addresses important procedures, such as point cloud processing, depth estimation, and multi-view stereo techniques. It emphasizes the benefits and drawbacks of each strategy while highlighting the necessity of combining various modalities for effective rebuilding. To improve reconstruction accuracy, context-aware algorithms and the inclusion of semantic data are also being investigated. Efforts are made to overcome concerns with scalability, ambiguous features, and occlusions in form reconstruction. Deep learning models' computational complexity is also examined, highlighting the necessity for effective algorithms that strike a compromise between accuracy and speed. In order to evaluate the effectiveness of reconstruction algorithms in an objective manner, the significance of benchmark datasets and assessment criteria is also emphasized.

Challenging requirements and optical depth estimation techniques in laparoscopy

Challenging requirements and optical depth estimation techniques in laparoscopy