Accuracy Of Depth Estimation Research Articles

Depth completion towards different sensor configurations via relative depth map estimation and scale recovery

Depth completion, which combines additional sparse depth information from the range sensors, substantially improves the accuracy of monocular depth estimation, especially using the deep-learning-based methods. However, these methods can hardly produce satisfactory depth results when the sensor configuration changes at test time, which is important for real-world applications. In this paper, the problem is tackled by our proposed novel two-stage mechanism, which decomposes depth completion into two subtasks, namely relative depth map estimation and scale recovery. The relative depth map is first estimated from a single color image with our designed scale-invariant loss function. Then the scale map is recovered with the additional sparse depth. Experiments on different densities and patterns of the sparse depth input show that our model always produces satisfactory depth results. Besides, our approach achieves state-of-the-art performance on the indoor NYUv2 dataset and performs competitively on the outdoor KITTI dataset, demonstrating the effectiveness of our method.

RDRF-Net: A pyramid architecture network with residual-based dynamic receptive fields for unsupervised depth estimation

Image depth estimation is a challenging problem in computer vision, especially considering both high accuracy and low run time. To save run time and maintain high accuracy, we present a new light-weight model in this paper, i.e., a Residual-based Dynamic Receptive Field Network (RDRF-Net). This model can automatically select the receptive fields suitable for different image scales to generate the depth maps with higher fitting degrees. Residual design and bottleneck layers are used to compress the network for reducing run time. Three groups of experiments are performed on the KITTI dataset to test the accuracy, computation time, and the impact of dynamic receptive fields. Experimental results show that RDRF-Net has comparable accuracy with Godard’s model and significantly outperforms it in terms of run time. In addition, it performs closely to Pyd-Net in terms of run time and beats Pyd-Net’s accuracy. Experiments also demonstrate the beneficial impact of dynamic receptive fields on improving depth estimation accuracy.

Depth estimation method of surface of micropart in microassembly space based on microscopic vision tomographic scanning images.

Three-dimensional (3D) morphology of microparts has an important influence on performance of microassembly system that mainly assembles microparts in millimetre and micron scale. Because 3D morphology of microparts cannot be accurately obtained by conventional microscopic vision system, a depth estimation method of surface of micropart in microassembly space based on microscopic vision tomographic scanning (MVTS) images is proposed in this paper. The proposed method uses the positions of pixels with the largest focus values in MVTS image to construct the isodepth contours of surface of micropart and obtains the depth values of micropart's surface at the positions of MVTS by assigning depth values to corresponding isodepth contours. The MVTS images are obtained by MVTS and pixels with the largest focus values in MVTS image are obtained by focus measurement of MVTS images of micropart in microassembly space. On these bases, 3D spatial interpolation method is applied to map depth value of space between adjacent isodepth contours and to obtain depth values of all surface of micropart. Simulation experiments are carried out to verify the proposed method by generating simulated MVTS image array from two simulation objects, and the influence parameters of the proposed method are analysed. In established experimental setup of microassembly that can realise MVTS, experimental verification for the proposed depth estimation method are carried out by using cone cavity and end jaws of microgripper. 3D morphologies of depth maps of cone cavity and end jaws of microgripper are registered with their respective CAD models using iterative nearest point registration algorithm to quantify accuracy of depth estimation. The research results show that 3D morphology of micropart can be obtained by the proposed method and has better accuracy than those by conventional shape from focus method. This method provides a new way to obtain the morphology of microparts and lays a foundation for improving the accuracy and efficiency of gripping, alignment and approaching microparts in microassembly systems.

Occlusion-aware light field depth estimation using side window angular coherence.

Depth estimation is crucial in many light field applications. However, the accuracy of light field depth estimation is prone to be affected by occlusions. In this paper, a method of side window angular coherence is proposed to handle different types of occlusions, and the ability of the proposed method to resist occlusions is theoretically analyzed. The angular patch is divided into several discrete side window subsets. These subsets are a pure occluder-type subset, a pure object point-type subset, and a hybrid-type subset. The photo-consistency of the pure object point-type subset can reflect the true depth. Meanwhile, the occlusion edges can be detected to identify occluded points and nonoccluded points so the robustness of the algorithm can be further enhanced by processing the two types of points. Moreover, fast guided filtering is applied to cost volume for improving the accuracy of depth estimation. Experimental results demonstrate that our method outperforms the state-of-the-art depth estimation methods on both synthetic and real scenes, especially near occlusion boundaries.

Fast Depth Prediction and Obstacle Avoidance on a Monocular Drone Using Probabilistic Convolutional Neural Network

Recent studies employ advanced deep convolutional neural networks (CNNs) for monocular depth perception, which can hardly run efficiently on small drones that rely on low/middle-grade GPU(e.g. TX2 and 1050Ti) for computation. In addition, the methods which can effectively and efficiently produce probabilistic depth prediction with a measure of model confidence have not been well studied. The lack of such a method could yield erroneous, sometimes fatal, decisions in drone applications (e.g. selecting a waypoint in a region with a large depth yet a low estimation confidence). This paper presents a real-time onboard approach for monocular depth prediction and obstacle avoidance with a lightweight probabilistic CNN (pCNN), which will be ideal for use in a lightweight energy-efficient drone. For each video frame, our pCNN can efficiently predict its depth map and the corresponding confidence. The accuracy of our lightweight pCNN is greatly boosted by integrating sparse depth estimation from a visual odometry into the network for guiding dense depth and confidence inference. The estimated depth map is transformed into Ego Dynamic Space (EDS) by embedding both dynamic motion constraints of a drone and the confidence values into the spatial depth map. Traversable waypoints are automatically computed in EDS based on which appropriate control inputs for the drone are produced. Extensive experimental results on public datasets demonstrate that our depth prediction method runs at 12Hz and 45Hz on TX2 and 1050Ti GPU respectively, which is 1.8X~5.6X faster than the state-of-the-art methods and achieves better depth estimation accuracy. We also conducted experiments of obstacle avoidance in both simulated and real environments to demonstrate the superiority of our method to the baseline methods.

A Novel Occlusion-Aware Vote Cost for Light Field Depth Estimation.

Capturing the directions of light by light field cameras powers next-generation immersive multimedia applications. A critical problem in taking advantage of the rich visual information in light field images is depth estimation. Conventional light field depth estimation methods build a cost volume that measures the photo-consistency of pixels refocused to a range of depths, and the highest consistency indicates the correct depth. This strategy works well in most regions but usually generates blurry edges in the estimated depth map due to occlusions. Recent work shows that integrating occlusion models to light field depth estimation can largely reduce blurry edges. However, existing occlusion handling methods rely on complex edge-aided processing and post-refinement, and this reliance limits the resultant depth accuracy and impacts on the computational performance. In this paper, we propose a novel occlusion-aware vote cost (OAVC) which is able to accurately preserve edges in the depth map. Instead of using photo-consistency as an indicator of the correct depth, we construct a novel cost from a new perspective that counts the number of refocused pixels whose deviations from the central-view pixel are less than a small threshold, and utilizes that number to select the correct depth. The pixels from occluders are thus excluded in determining the correct depth. Without the use of any explicit occlusion handling methods, the proposed method can inherently preserve edges and produces high-quality depth estimates. Experimental results show that the proposed OAVC outperforms state-of-the-art light field depth estimation methods in terms of depth estimation accuracy and computational complexity.

Iterative Refinement of Uniformly Focused Image Set for Accurate Depth from Focus

Estimating the 3D shape of a scene from differently focused set of images has been a practical approach for 3D reconstruction with color cameras. However, reconstructed depth with existing depth from focus (DFF) methods still suffer from poor quality with textureless and object boundary regions. In this paper, we propose an improved depth estimation based on depth from focus iteratively refining 3D shape from uniformly focused image set (UFIS). We investigated the appearance changes in spatial and frequency domains in iterative manner. In order to achieve sub-frame accuracy in depth estimation, optimal location of focused frame in DFF is estimated by fitting a polynomial curve on the dissimilarity measurements. In order to avoid wrong depth values on texture-less regions we propose to build a confidence map and use it to identify erroneous depth estimations. We evaluated our method on public and our own datasets obtained from different types of devices, such as smartphones, medical, and normal color cameras. Quantitative and qualitative evaluations on various test image sets show promising performance of the proposed method in depth estimation.

Snow Depth Variations in Svalbard Derived from GNSS Interferometric Reflectometry

Snow plays a critical role in hydrological monitoring and global climate change, especially in the Arctic region. As a novel remote sensing technique, global navigation satellite system interferometric reflectometry (GNSS-IR) has shown great potential for detecting reflector characteristics. In this study, a field experiment of snow depth sensing with GNSS-IR was conducted in Ny-Alesund, Svalbard, and snow depth variations over the 2014–2018 period were retrieved. First, an improved approach was proposed to estimate snow depth with GNSS observations by introducing wavelet decomposition before spectral analysis, and this approach was validated by in situ snow depths obtained from a meteorological station. The proposed approach can effectively separate the noise power from the signal power without changing the frequency composition of the original signal, particularly when the snow depth changes sharply. Second, snow depth variations were analyzed at three stages including snow accumulation, snow ablation and snow stabilization, which correspond to different snow-surface-reflection characteristics. For these three stages of snow depth variations, the mean absolute errors (MAE) were 4.77, 5.11 and 3.51 cm, respectively, and the root mean square errors (RMSE) were 6.00, 6.34 and 3.78 cm, respectively, which means that GNSS-IR can be affected by different snow surface characteristics. Finally, the impact of rainfall on snow depth estimation was analyzed for the first time. The results show that the MAE and RMSE were 2.19 and 2.08 cm, respectively, when there was no rainfall but 5.63 and 5.46 cm, respectively, when it was rainy, which indicates that rainfall reduces the accuracy of snow depth estimation by GNSS-IR.

Deep eyes: Joint depth inference using monocular and binocular cues

Human visual system relies on both monocular focusness cues and binocular stereo cues to gain effective 3D perception. Correspondingly, depth from focus/defocus (DfF/DfD) and stereo matching are two most studied passive depth sensing schemes, which are traditionally solved in separate tracks. However, the two techniques are essentially complementary: the monocular cue from DfF/DfD can robustly handle repetitive textures and occlusion that are problematic for stereo matching whereas the binocular cue from stereo matching is insensitive to defocus blurs and can resolve large depth range. In this paper, we emulate human perception and present unified learning-based techniques to conduct hybrid DfF/DfD and stereo matching. We first construct a comprehensive focal stack dataset synthesized by depth-guided light field rendering. Next, we propose different network architectures to suit various inputs, including focal stack, stereo image pair, binocular focal stack, a focus-defocus image pair and defocus-stereo image triplet. We also exploit different connection methods between the separate networks for integrating them into an optimized solution to produce high fidelity disparity maps. For experiment, we further explore different hardware setup to capture both monocular and binocular depth cues. Results show that our new learning-based hybrid techniques can significantly improve accuracy and robustness in depth estimation.

Unsupervised-Learning-Based Continuous Depth and Motion Estimation With Monocular Endoscopy for Virtual Reality Minimally Invasive Surgery

Three-dimensional display and virtual reality technology have been applied in minimally invasive surgery to provide doctors with a more immersive surgical experience. One of the most popular systems based on this technology is the Da Vinci surgical robot system. The key to build the in vivo 3-D virtual reality model with a monocular endoscope is an accurate estimation of depth and motion. In this article, a fully unsupervised learning method for depth and motion estimation using the continuous monocular endoscopic video is proposed. After the detection of highlighted regions, EndoMotionNet and EndoDepthNet are designed to estimate ego-motion and depth, respectively. The timing information between consecutive frames is considered with a long short-term memory layer by EndoMotionNet to enhance the accuracy of ego-motion estimation. The estimated depth value of the previous frame is used to estimate the depth of the next frame by EndoDepthNet with a multimode fusion mechanism. The custom loss function is defined to improve the robustness and accuracy of the proposed unsupervised-learning-based method. Experiments with the public datasets verify that the proposed unsupervised-learning-based continuous depth and motion estimation method can effectively improve the accuracy of depth and motion estimation, especially after processing the frame.

Monocular depth estimation based on deep learning: An overview

Depth information is important for autonomous systems to perceive environments and estimate their own state. Traditional depth estimation methods, like structure from motion and stereo vision matching, are built on feature correspondences of multiple viewpoints. Meanwhile, the predicted depth maps are sparse. Inferring depth information from a single image (monocular depth estimation) is an ill-posed problem. With the rapid development of deep neural networks, monocular depth estimation based on deep learning has been widely studied recently and achieved promising performance in accuracy. Meanwhile, dense depth maps are estimated from single images by deep neural networks in an end-to-end manner. In order to improve the accuracy of depth estimation, different kinds of network frameworks, loss functions and training strategies are proposed subsequently. Therefore, we survey the current monocular depth estimation methods based on deep learning in this review. Initially, we conclude several widely used datasets and evaluation indicators in deep learning-based depth estimation. Furthermore, we review some representative existing methods according to different training manners: supervised, unsupervised and semi-supervised. Finally, we discuss the challenges and provide some ideas for future researches in monocular depth estimation.

Detection and localization of rebar in concrete by deep learning using ground penetrating radar

Ground penetrating radar (GPR) has been widely used for detection and localization of reinforced steel bar (rebar) in concrete in a non-destructive way. However, manual interpretation of a large number of GPR images is time-consuming, and the results highly depend on practitioner experience and the available priori information. This paper proposes an automatic detection and localization method using deep learning and migration. Firstly, a Single Shot Multibox Detector (SSD) model is established to identify regions of interest containing hyperbolas in a GPR image. This deep learning model is trained using a real GPR dataset, which contains 13,026 rebar targets in 3992 images, collected on residential buildings under construction. Secondly, each target region is migrated and transformed into a binary image to locate the rebar. After the binarization, the apex of the focused cluster is obtained and used to estimate both the horizontal position and the depth of the rebar. The testing results show that the detection accuracy of the proposed artificial intelligence method is 90.9%. The computation time needed for processing a GPR image with a size of 300 × 300 pixels is only 0.47 s. The depth estimation error in a laboratory experiment is <1.5 mm (5%), and the lateral position error is <0.7 cm. Therefore, it is concluded that the proposed method can automatically detect the rebar from GPR images in real time when a handheld GPR system is operated at a walking speed and the depth estimation accuracy is acceptable in practice.

Semi-Supervised Monocular Depth Estimation Based on Semantic Supervision

Monocular depth estimation by unsupervised learning is a potential strategy, which is mainly self-supervised by calculating view reconstruction loss from stereo pairs or monocular sequences. However, most existing works only consider the geometric information during training, without using semantics. We propose a semantic monocular depth estimation (SE-Net), a neural network framework that estimates depth using semantic information and video sequences. The whole framework is semi-supervised, because we take advantage of labelled semantic ground truth data. In view of the structural consistency between the semantically segmented image and the depth map, we first perform semantic segmentation on the image, and then use the semantic labels to guide the construction of the depth estimation network. Experiments on the KITTI dataset show that learning semantic information from images can effectively improve the effect of monocular depth estimation, and SE-Net is superior to the most advanced methods in depth estimation accuracy.

Depth Estimation of Hard Inclusions in Soft Tissue by Autonomous Robotic Palpation Using Deep Recurrent Neural Network

Accurately detecting tumors and estimating the depth of tumors is essential in the surgical removal of tumors. In robotic-assisted surgery, autonomous robotic palpation has the potential to provide more precise detection, tumors’ depth estimation, and less intrusion when normal tissues surround tumors. In this article, by mimicking the human finger touch, we propose a tactile sensing-based deep recurrent neural network (DRNN) with long short-term memory (LSTM) architecture to improve the accuracy of the detection and depth estimation of tumors embedded in soft tissue. In the experimental setup, the hard inclusions simulate the tumors, while the phantom tissue is fabricated by silicon to simulate the soft tissue. During the experiment, the data from the force sensor and displacement of the robot palpation probe are for detection and depth estimation purposes. The collected sequential data set of the force and the displacement of the probe during one completed palpation process will go through the proposed DRNN network with deep LSTM architecture, in which the temporal dependencies of the sequential data will be captured in the cell states in the deep LSTM layers. Subsequently, the softmax classifier is adopted to determine if there is any hard inclusion exists and offer the depth estimation of the hard inclusions. Experiments based on 396 real data sets demonstrate that the detection accuracy for the testing data set is 99.2% and the depth estimation accuracy for the testing data set is 95.8%. The accuracy of the proposed method is best when comparing with other widely used methods. Note to Practitioners —The palpation of tumors motivated this article in the robot-assisted surgical systems through tactile feedback. In order to mimic the human touch on the soft tissue, this article presents a deep-learning-based approach to estimate the depth of the hard inclusions in the phantom tissue through force information. The displacement of the palpation probe and the touch force during one palpation are recorded as data sequences to train the deep model, which aims to capture dynamics and long-term dependence of the palpation process. In this article, we made the first successful attempt to accurately estimate the depth of the hard inclusions buried at different locations of the phantom tissue using only force information. The proposed approach can work in different robot-assisted scenarios, such as master–slave robotic surgery. In the clinic applications, the force sensor will be integrated at the end-effector of the robotic manipulator. According to the specific requirements, the force sensor and the robotic manipulator might be different from those used in this article. For some applications, such as the laparoscopic interventions, the complete vertical contact tends to be difficult to obtain due to the laparoscopic port effects. The projection of the recorded force data and displacement can obtain the information in the normal direction. The future work is going to be extended to tissue environments with arbitrary surface and tumors with various shapes/depths for more complex and prospective clinical applications.

Preprocedural Ultrasound Estimates of Epidural Depth: Transverse Median Plane is Comparable to Paramedian Sagittal Oblique Plane in Non-Pregnant Patients

ObjectiveThe aim of this study was to assess the accuracy and precision of the actual epidural depth (ND) and ultrasound (US) estimates of the distance from the skin to the epidural space in the transverse median plane (ED/TM) and paramedian sagittal oblique plane (ED/PSO) in patients who were not pregnant and who were undergoing unilateral inguinal hernia repair surgery under combined spinal epidural anaesthesia.MethodsA total of 100 patients with the American Society of Anaesthesiology status I–III of either gender were recruited. All epidurals were inserted at the L3–4 intervertebral space, and the epidural needle was marked with a sterile marker. The ND was measured between the marker and the tip of the needle. The ED/TM and ED/PSO were assessed with a 2–5 MHz curved array probe at the L3–4 intervertebral space. The Pearson correlation coefficient and Lin’s concordance correlation coefficient were used to assess the precision and the Bland–Altman test was used to assess the 95% limit of agreement.ResultsThe ND was 48.98±6.91 mm, the ED/PSO was 48.92±6.91 mm and the ED/TM was 48.90±6.91 mm. The Pearson correlation coefficient between ND and ED/PSO was 0.995 (p<0.001) and 0.994 (p<0.001) with ED/TM. Lin’s concordance correlation coefficient for ND and ED/PSO was 0.995 (95%CI [confidence interval]: 0.992–0.997), and 0.993 for ND and ED/TM (95% CI: 0.990–0.996). The 95% limit of agreement between ND and ED/PSO was 0.70–1.37, and for ND and ED/TM, it was 0.79–1.54.ConclusionPreprocedural ultrasound (US) scanning in both planes provides reliable estimates for actual epidural depth in non-pregnant patient population.

Integrated Autonomous Relative Navigation Method Based on Vision and IMU Data Fusion

An integrated autonomous relative navigation method based on vision and IMU data fusion was proposed in this paper, which can improve the position accuracy effectively and has strong adaptability to environmental changes. Firstly, IMU pre-integration formula based on Runge Kutta method was derived, which can improve the pre-integration position accuracy and reduce the accumulated error effectively. Secondly, an inverse depth estimation method based on the mixed probability model was proposed during the system initialization process, which can improve the accuracy of camera depth estimation and provide better initial conditions for back-end optimization. Thirdly, a sliding window filtering method based on the probability graph was proposed, which can avoid repeated calculations and improve the sliding window filtering efficiency. Forthly, combined with the advantages of the direct method and the feature point method, a mixed re-projection optimization method was proposed, which can expand the application scope of the method and improve the optimization accuracy effectively. Finally, in the closed-loop optimization, a closed-loop optimization method based on similar transformation is proposed to eliminate the accumulated error. In order to verify the environmental adaptability of the method and the impact of closed-loop detection on the relative navigation system, indoor and outdoor experiments were carried out with a hand-held camera and an IMU. EuRoC dataset was used in the experiments and the proposed method was compared with some classical methods. The experimental results showed that this method has high accuracy and robustness.

Unsupervised Monocular Training Method for Depth Estimation Using Statistical Masks

Recently, unsupervised monocular training methods based on convolutional neural networks have already shown surprisingly progress in improving the accuracy of depth estimation. However, the performance of these methods suffers deeply from problematic pixels such as occluded pixels, low-texture pixels, and so on. In this paper, we introduce a method to a mask by the statistic of error maps for segmenting the problematic pixels. Different from the conventional methods which use additional segmentation networks to classify problematic pixels, we use a multi-task learning architecture to generate identical mask, mean mask, and variance mask for filtering the problematic pixels. Experimental results show that our proposed method has satisfactory performance compared with other relative methods on the KITTI dataset. Moreover, we also apply our method to the UAV dataset VisDrone, and the results also indicate the effectiveness of the method in detecting moving objects.

Linear depth inversion sensitivity to wave viewing angle using synthetic optical video

The accuracy of bathymetry estimated by optical implementations of remotely sensed depth inversion algorithms is in part related to the presence of optical wave signal in the images, which depend nonlinearly on the water surface slope. The signal to noise ratio in video images of waves decreases under large azimuthal angles between the camera and wave propagation direction, which can result in poor bathymetry estimation. We quantified errors in depth estimation by analysing the sensitivity of the optical implementation of cBathy v1.1, a widely applied algorithm for depth inversion in coastal regions, to wave viewing angle using synthetic tests. We found relative root mean square errors between 0.02 and 0.08 when the azimuthal angle between the camera look direction and wave approach was less than 75°. However, for higher azimuthal angles, the wave signal was dominated by short wavelengths in the optical images leading to larger depth errors (relative root mean square error = 0.2). We also investigated the sensitivity of the initial guess of the wave direction in the nonlinear solution used by the cBathy v1.1 algorithm to estimate water depth. Observed water depth errors caused by wave viewing angle or initial guess of the wave direction are shown in part to be related to errors in the estimates of frequency and wavenumber. The synthetic methodology and the results of the sensitivity analysis can be generalized to test the accuracy of depth estimation in shore-based video monitoring systems, to design future fixed camera coastal video monitoring stations or to drive the choice of the better viewing angles using small Unmanned Aerial Systems (sUAS) using the Matlab Toolbox we developed.

Evaluation of nonparametric tree models for predicting the scour depth of bridge piers

One of the main causes of damage to bridges, especially during flood event is the scour around the bridge. Determination of the depth of scour around the bridge piers plays a very important role in designing the bridges against this destructive factor. The complexity of the bridge scour and the effects of different parameters on its estimation more clearly reveal the necessity of using a nonlinear and comprehensive model in this field. In this present study, decision tree models, as nonparametric models, are used to estimate the scour depth. Furthermore, the statistics of different bridges and four tree methods are used. The data used to train and test decision trees including flow the velocity of upstream, the median grain size, flow depth, and the pier width, the skew of the pier to approach flow, the length of the pier, the grain size of bed material for which 84 percent is finer, a multiplying factor, input variables, and the depth of scour as output in the model. 75% of the available data is used for model training and the remaining 25% for testing. The results show that among the four models (CART, C5, QUEST, CHAID) examined, C5 model, considering the comparison of the root mean square error parameters and the coefficient of determination, is more accurate in computing the scour depth of the bridge, the amount coefficients of determination in this model is in training and testing steps are 0.92 and 0.76, as well as the mean square error values of the error is 0.56 and 0.72 respectively. Furthermore, the results reveal the QUEST model does not have a proper accuracy in scour depth estimation. Furthermore, the analysis of the models shows flow depth, the flow velocity in the upstream have the greatest effect on the scour depth.

Depth Estimation with Ego-Motion Assisted Monocular Camera

We propose a method to estimate the distance to objects based on the complementary nature of monocular image sequences and camera kinematic parameters. The fusion of camera measurements with the kinematics parameters that are measured by an IMU and an odometer is performed using an extended Kalman filter. Results of field experiments with a wheeled robot corroborated the results of the simulation study in terms of accuracy of depth estimation. The performance of the approach in depth estimation is strongly affected by the mutual observer and feature point geometry, measurement accuracy of the observer’s motion parameters and distance covered by the observer. It was found that under favorable conditions the error in distance estimation can be as small as 1% of the distance to a feature point. This approach can be used to estimate distance to objects located hundreds of meters away from the camera.