Accurate Depth Estimation Research Articles

LE-VINS: A Robust Solid-State-LiDAR-Enhanced Visual-Inertial Navigation System for Low-Speed Robots

Accurate and long-distance depth estimation for visual landmarks is challenging in visual-inertial navigation systems (VINS). In visual-degenerated scenes with illumination changes, moving objects, or weak texture, depth estimation may be more difficult, resulting in poor robustness and accuracy. For low-speed robot navigation, we present a solid-state-LiDAR-enhanced VINS (LE-VINS) to improve the system robustness and accuracy in challenging environments. The point clouds from the solid-state LiDAR are projected to the visual keyframe with the inertial navigation system (INS) pose for depth association while compensating for the motion distortion. A robust depth-association method with an effective plane-checking algorithm is proposed to estimate the landmark depth. With the estimated depth, we present a LiDAR depth factor to construct accurate depth measurements for visual landmarks in factor graph optimization (FGO). The visual feature, LiDAR depth, and IMU measurements are tightly fused within the FGO framework to achieve maximum-a-posterior estimation. Field tests were conducted on a low-speed robot in large-scale challenging environments. The results demonstrate that the proposed LE-VINS yields significantly improved robustness and accuracy compared to the original VINS. Besides, LE-VINS exhibits superior accuracy than the state-of-the-art LiDAR-visual-inertial navigation system. LE-VINS also outperforms the existing LiDAR-enhanced method, benefiting from the robust depth-association algorithm and the effective LiDAR depth factor.

Predictive explicit expressions from data-driven models for estimation of scour depth below ski-jump bucket spillways

Abstract Scour depth estimation is an essential factor in water-related engineering problems. Scouring below spillways may endanger a dam's stability and even lead to dam destruction. As a result, it has undesirable environmental effects due to dam failure. Hence, reliable and accurate scour depth estimation below spillways is an exciting topic for researchers. For this purpose, the published and reliable prototype data related to scour depth below ski jump bucket spillways (Ds) was used to develop data-driven models. This study employed two widely used decision tree (DT) methods, including the M5 model tree (M5MT) and the classification and regression tree (CART), and also multivariate adaptive regression splines (MARS) for the estimation of (Ds). The proposed methods provided explicit and clear equations with straightforward applications for estimating scour depth. For the quantitative assessments of the developed formulas, three common statistical metrics, namely root mean square error (RMSE), mean absolute error (MAE), and correlation coefficient (CC), were used. Moreover, comparison results with previous approaches existing in the literature indicated the efficacy of the suggested methods. The obtained results revealed that the MARS technique was the best approach for the estimation of scour depth.

Normal Assisted Pixel-Visibility Learning With Cost Aggregation for Multiview Stereo

Multiple-View Stereo (MVS) aims to reconstruct the dense 3D representations of scenes. MVS has potential applications in the fields of autonomous driving (unstructured environment construction) and robotic navigation (visual-inertial navigation). To mitigate the error of depth estimation in low-textured or occluded regions, this work proposes a two-stage multi-view stereo network for fast and accurate depth estimation. The improvements of this work over the state of the art are as follows: 1) Sparse costs are constructed to jointly predict the initial depth map and surface normal by cost regularization, which proves that the surface normals can be estimated in this way with low memory consumption. 2) A new edge refinement block is developed to refine the coarse surface normal to obtain a fine-grained surface normal map. 3) Instead of using the general variance-based metric to equally aggregate cost, a new content-adaptive cost aggregation mechanism based on the similarity of the neighboring surface normal is designed for reliable cost aggregation. To the best of our knowledge, the proposed work is the first trainable network that leverages surface normal as guidance to capture neighboring pixel-visibility, which is an effective supplement to existing depth/normal estimation frameworks. Experimental results indicate that our method can not only achieve accurate depth estimation for scene perception but also make no concession to the real-time performance and limited memory bottleblock. Multiple-view stereo (MVS) aims to reconstruct the dense 3D representations of scenes. It is widely used in the fields of industrial measurement, autonomous driving, and robotic navigation. To mitigate the error of depth estimation in challenging scenarios, this work proposes a two-stage multi-view stereo network for fast and accurate depth estimation. Our method is the first trainable network that leverages surface normal as pixel-visibility guidance to aggregate reliable cost, which could achieve accurate depth estimation and provide the perception ability for the robot. The proposed method has great potential in the fields of 3D reconstruction, industrial measurement, and robotic navigation to estimate real-time and accurate depth with limited memory consumption.

Hybrid model of support vector regression and innovative gunner optimization algorithm for estimating ski-jump spillway scour depth

Scour hole that occurs downstream of the hydraulic structures threatens the safety and stability of the hydraulic structures. The scour around the structures is a complex and important hydraulic phenomenon; hence, it requires a data extensive research for the accurate estimation of scour depth. Although many analytical models are available for scour depth estimation, they suffer from huge limitations. In this research, the support vector regression (SVR) model and SVR ensemble with the metaheuristic algorithm of innovative gunner (SVR-AIG) models have been developed for accurate prediction of scour depth downstream of the ski-jump spillways. Field measurements including head and discharge intensity are used for developing the models. The performances of the models are compared using root mean square error (RMSE), mean average error (MAE), and correlation coefficient (CC) criteria and some statistical plots. The results showed that the hybrid SVR-AIG-based estimations (with CC = 0.987, 0.991, RMSE = 2.839, 1.987, and MAE = 2.247, 1.201) are more accurate than the SVR standalone model estimations (with CC = 0.942, 0.975, RMSE = 5.686, 4.040, and MAE = 4.114, 3.201) at the training and testing phases. This study is an important reference for analyzing the high capability of the AIG as an optimization tool in improving scour estimations of a standalone model. Also, this algorithm eliminates the trial-and-error procedure to optimize the internal parameters during the model development.Graphical abstract

Visual Attention-Based Self-Supervised Absolute Depth Estimation Using Geometric Priors in Autonomous Driving

Although existing monocular depth estimation methods have made great progress, predicting an accurate absolute depth map from a single image is still challenging due to the limited modeling capacity of networks and the scale ambiguity issue. In this paper, we introduce a fully Visual Attention-based Depth (VADepth) network, where spatial attention and channel attention are applied to all stages. By continuously extracting the dependencies of features along the spatial and channel dimensions over a long distance, VADepth network can effectively preserve important details and suppress interfering features to better perceive the scene structure for more accurate depth estimates. In addition, we utilize geometric priors to form scale constraints for scale-aware model training. Specifically, we construct a novel scale-aware loss using the distance between the camera and a plane fitted by the ground points corresponding to the pixels of the rectangular area in the bottom middle of the image. Experimental results on the KITTI dataset show that this architecture achieves the state-of-the-art performance and our method can directly output absolute depth without post-processing. Moreover, our experiments on the SeasonDepth dataset also demonstrate the robustness of our model to multiple unseen environments.

29.5: Enhanced depth of field of integral imaging display using bifocal microlens array fabricated by two‐step lithography

Due to the limitation of traditional microlens arrays (MLAs) in integral imaging display, the depth of field (DOF) is restricted in space and the center depth plane are difficult to extended in a large range. Here we propose a microfabrication method based on bifocal MLAs to improve DOF. The bifocal MLAs for extended DOF was fabricated by using two‐step photolithography and thermal reflow. This method allows diverse microlenses of high to low numerical aperture to achieve high spatial resolution as well as accurate depth estimation. Microlenses of different focal lengths were simultaneously deposited on a substrate by repeated photolithography with multiple photomasks with alignment mark to define micro‐posts of different thicknesses. Hexagonally packaged bifocal MLAs clearly show the DOF extended from 0.004 mm to 4.908 mm for 57.6 �m in lens diameter and their corresponding object distance ranges from 0.125 mm to 0. 165 mm. Based on the proposed scheme, this method provides potential applications in integral imaging 3D display or light field display.

Real-Time Self-Supervised Monocular Depth Estimation Without GPU

Single-image depth estimation represents a longstanding challenge in computer vision and although it is an ill-posed problem, deep learning enabled astonishing results leveraging both supervised and self-supervised training paradigms. State-of-the-art solutions achieve remarkably accurate depth estimation from a single image deploying huge deep architectures, requiring powerful dedicated hardware to run in a reasonable amount of time. This overly demanding complexity makes them unsuited for a broad category of applications requiring devices with constrained resources or memory consumption. To tackle this issue, in this paper a family of compact, yet effective CNNs for monocular depth estimation is proposed, by leveraging self-supervision from a binocular stereo rig. Our lightweight architectures, namely PyD-Net and PyD-Net2, compared to complex state-of-the-art trade a small drop in accuracy to drastically reduce the runtime and memory requirements by a factor ranging from <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$100\times $ </tex-math></inline-formula> . Moreover, our networks can run real-time monocular depth estimation on a broad set of embedded or consumer devices, even not equipped with a GPU, by early stopping the inference with negligible (or no) loss in accuracy, making it ideally suited for real applications with strict constraints on hardware resources or power consumption.

Lightweight Depth Completion Network with Local Similarity-Preserving Knowledge Distillation.

Depth perception capability is one of the essential requirements for various autonomous driving platforms. However, accurate depth estimation in a real-world setting is still a challenging problem due to high computational costs. In this paper, we propose a lightweight depth completion network for depth perception in real-world environments. To effectively transfer a teacher’s knowledge, useful for the depth completion, we introduce local similarity-preserving knowledge distillation (LSPKD), which allows similarities between local neighbors to be transferred during the distillation. With our LSPKD, a lightweight student network is precisely guided by a heavy teacher network, regardless of the density of the ground-truth data. Experimental results demonstrate that our method is effective to reduce computational costs during both training and inference stages while achieving superior performance over other lightweight networks.

Occlusion-aware light field depth estimation with view attention

Depth estimation for light field images is crucial in light field applications such as image-based rendering and refocusing. Previous learning-based methods com bining neural network with cost volume can achieve accurate depth estimation but fail in regions with occlusion. In this paper, a two-stage attention-based occlusion-aware light field depth estimation network is proposed. In the initial depth estimation stage, the sub-aperture images are divided into four groups based on different view directions and four initial cost volumes are constructed using the feature maps from each group to aggregate initial depth maps. Then in the refined depth estimation stage, the four aggregated volumes from the initial stage are fused into one based on the view attention, where features of views with less occlusion are highly weighted to provide more effective information. Experiment results demonstrate that the proposed method can accomplish robust and accurate depth estimation in the presence of occlusion, which ranks the first place on 4D light field benchmark in terms of most accuracy metrics.

Underwater self-supervised depth estimation

Accurate underwater depth estimation is a cornerstone of reaching autonomous underwater exploration. However, it is incredibly tricky due to the inherent attenuation character and heavy noise. Fortunately, the depth-changing trend and underwater light attenuation are closely correlated, providing powerful clues for underwater depth estimation. Rather than simulating the underwater attenuation through formulas, we propose an underwater self-supervised depth estimation neural network in our work. With the guidance of multiple constraints, which are meticulously designed based on the comprehensive analyses of underwater characters, this network can learn the depth-changing trend by itself from attenuation information in underwater monocular videos. Our detailed experiments on underwater datasets prove that the proposed framework can obtain accurate and fine-grained depth maps. We believe the work may provide an economical solution for underwater perception.

PCNet: Paired channel feature volume network for accurate and efficient depth estimation

Most state-of-the-art deep learning based depth estimation methods follow the pipeline of firstly forming a 4D cost volume (feature dimension, max disparity, height, and width) and then regressing disparity from the cost volume by several 3D convolutional layers. Applying 3D operations on the 4D tensor leads to unacceptable computational complexity and memory cost. To solve the problem, we aim at replacing the 4D cost volume with 3D cost volume so that the disparity can be regressed by 2D convolutions to achieve a good balance between efficiency and effectiveness. To this end, a light-weighted network, called PCNet, is proposed to generate 3D cost volume. The main novelty lies in the proposed Paired Channel Feature Volume (PCFV) which is capable of combining the features of stereo pairs with specially designed 3D filters to preliminarily encode the relationship between each pair of the channels. Moreover, a densely connected aggregation on the outputs of PCFV is performed to exploit much richer contextual information. Experimental results on the SceneFlow, KITTI 2012, and KITTI 2015 datasets demonstrate that the proposed PCNet achieves comparable accuracy with state-of-the-art methods and keeps high efficiency as well.

Interannual and seasonal variations of permafrost thaw depth on the Qinghai-Tibetan Plateau: A comparative study using long short-term memory, convolutional neural networks, and random forest.

An accurate estimation of thaw depth is critical to understanding permafrost changes due to climate warming on the Qinghai-Tibetan Plateau (QTP). However, previous studies mainly focused on the interannual changes of active layer thickness (ALT) across the QTP, and little is known about the changes in the seasonal thaw depth. Machine learning (ML) is a critical tool to accurately estimate the ALT of permafrost, but a direct comparison of ML with deep learning (DL) in ALT projection regarding the model performance is still lacking. Here, ML, namely random forest (RF), and DL algorithms like convolutional neural networks (CNN) and long short-term memory (LSTM) neural networks were compared to estimate the interannual changes of ALT and seasonal thaw depth on the QTP. Meteorological series, in-situ collected ALT observations, and geospatial information were used as predictors. The results show that both ML and DL methods are capable of estimating ALT and seasonal thaw depth in permafrost areas. The CNN and LSTM models developed using longer lagging times exhibit better performance in thaw depth prediction while the RF models are either mediocre or sometimes even worse as the lagging time increases. The results show that the ALT from 2003 to 2011 on the QTP exhibits an increasing trend, especially in the northern region. In addition, 68.8%, 88.7%, 52.5%, and 47.5% of the permafrost regions on the QTP have deepened seasonal thaw depth in spring, summer, autumn, and winter, respectively. The correlation between air temperature and permafrost thaw depth ranges from 0.65 to 1 with the time lag ranging from 1 to 32 days. This study shows that ML and DL can be effectively used in retrieving ALT and seasonal thaw depth of permafrost, and could present an efficient way to figure out the interannual and seasonal variations of permafrost conditions under climate warming.

Prediction of scour depth downstream of the flip bucket with machine learning techniques

One of the most common structures used for energy dissipation is the flip bucket. A jet passing through these spillways, after being thrown into the air and hitting the downstream bed, still has high energy, causing scour downstream of the spillway. Therefore, accurate estimation of the scour depth is important in the proper design of the main and related structures. In recent years, the use of computational intelligence has been widely used to estimate the scour depth accurately. In this research, the maximum scour depth was estimated using the three techniques of ‘gradient boosting decision tree’ (GBDT), ‘extra trees’ and ‘random forest’, then compared with previous results. The results indicate that the GBDT method, with an R2 of 0.992, a root mean square error of 0.231 and mean absolute error of 0.180, has the highest accuracy and lowest error.

Evaluation of Bromelain-Based Enzymatic Debridement Combined with Laser Doppler Imaging and Healing of Burn Wounds.

BackgroundAccurate estimation of burn depth is crucial for correct treatment decision making. Bromelain-based enzymatic debridement (ED) may improve clinical assessment of burn depth. Laser Doppler imaging (LDI) provides a valuable indicator of burn depth by analyzing microcirculation within tissue beds. This study aimed to evaluate bromelain-based enzymatic debridement combined with laser Doppler imaging and healing of 42 wounds in 19 patients with mixed second- and third-degree thermal burns.Material/MethodsWe included 42 wounds in 19 patients with mixed deep dermal and full-thickness thermal burns. All patients were treated with eschar-specific removal agent for ED. The perfusion of each wound after ED was assessed using LDI. Healing time was estimated by 2 experienced burn surgeons and marked by the observation of epithelization. The usefulness of the LDI performed after ED in predicting healing time was estimated. The findings were analyzed to determine a cut-off value for LDI that indicates if a burn will heal spontaneously.ResultsWe observed that burn wounds with higher mean perfusion healed faster. The analysis showed a strong relationship between perfusion after ED and healing time (Spearman rank correlation coefficient=−0.803). A mean perfusion greater than 296.89 indicated that the wound could heal spontaneously and does not require skin grafting.ConclusionsLDI examination of an already debrided wound allows for a reliable assessment of perfusion at an early stage of treatment. The use of a safe and effective debridement method in conjunction with a non-invasive diagnostic tool could improve burn management.

Sparse prior guided deep multi-view stereo

Recently, the learning-based multi-view stereo (MVS) methods have achieved notable progress. Despite this, the cost volumes of those methods are not robust enough for depth inference with the existence of noise, outlier and occlusion. In this work, we aim to reduce the matching ambiguities and increase the discrimination of the cost volume. For this purpose, we first propose a sparse prior guidance strategy which incorporates the sparse points of structure-from-motion (SfM) into cost volume construction. Further, we introduce a geometry-aware regularization module to enhance the representative power of cost volume regularization, which could adaptively fit the local geometric shapes. The two modules are straightforward yet effective, resulting in robust and accurate depth estimation. The detailed experiments are conducted on two public datasets (DTU and Tanks & Temples). The experimental results show that with the two components, the representative methods (e.g MVSNet and CasMVSNet) could make a significant gain in reconstruction quality.

A comparative study on equilibrium scour volume around circular cylinders in clay–sand mixed cohesive beds, at near threshold velocity of sand – an experimental approach

Abstract Local scour around a bridge pier is a complex phenomenon resulting from the interaction of the three-dimensional turbulent flow field around hydraulic structures. An accurate estimation of scour depth below the stream-bed is important during design since it determines the foundation levels and the expansion of the bridge foundation structures. The present study reveals the results of flume experiments on equilibrium scour depth and scour volumes around circular bridge pier models in clay–sand sediment mixtures. The scouring process in a cohesive sediment mixture is a complex interaction between clay–sand network structure and bed shear stresses. The bed shear stress reduces inside the scour hole as scour depth increases, and this is related to the different modes of scouring in clay–sand mixtures. An exponential equation for the non-dimensional scour volume is proposed considering all the experimental runs to get a specific understanding of the surrounding volume of scour hole from maximum equilibrium scour depth in a clay–sand mixed cohesive bed, when the approach flow velocity is near to the critical velocity for mixed sand. The proposed equations are validated using the pre-existing data from the literature dealing with experimental investigations on bridge scour using clay–sand mixed cohesive sediment and show good agreement with observed data.

Depth estimation from a single SEM image using pixel-wise fine-tuning with multimodal data

To support the ongoing size reduction in integrated circuits, the need for accurate depth measurements of on-chip structures becomes increasingly important. Unfortunately, present metrology tools do not offer a practical solution. In the semiconductor industry, critical dimension scanning electron microscopes (CD-SEMs) are predominantly used for 2D imaging at a local scale. The main objective of this work is to investigate whether sufficient 3D information is present in a single SEM image for accurate surface reconstruction of the device topology. In this work, we present a method that is able to produce depth maps from synthetic and experimental SEM images. We demonstrate that the proposed neural network architecture, together with a tailored training procedure, leads to accurate depth predictions. The training procedure includes a weakly supervised domain adaptation step, which is further referred to as pixel-wise fine-tuning. This step employs scatterometry data to address the ground-truth scarcity problem. We have tested this method first on a synthetic contact hole dataset, where a mean relative error smaller than 6.2% is achieved at realistic noise levels. Additionally, it is shown that this method is well suited for other important semiconductor metrics, such as top critical dimension (CD), bottom CD and sidewall angle. To the extent of our knowledge, we are the first to achieve accurate depth estimation results on real experimental data, by combining data from SEM and scatterometry measurements. An experiment on a dense line space dataset yields a mean relative error smaller than 1%.

Improving 360 Monocular Depth Estimation via Non-local Dense Prediction Transformer and Joint Supervised and Self-Supervised Learning

Due to difficulties in acquiring ground truth depth of equirectangular (360) images, the quality and quantity of equirectangular depth data today is insufficient to represent the various scenes in the world. Therefore, 360 depth estimation studies, which relied solely on supervised learning, are destined to produce unsatisfactory results. Although self-supervised learning methods focusing on equirectangular images (EIs) are introduced, they often have incorrect or non-unique solutions, causing unstable performance. In this paper, we propose 360 monocular depth estimation methods which improve on the areas that limited previous studies. First, we introduce a self-supervised 360 depth learning method that only utilizes gravity-aligned videos, which has the potential to eliminate the needs for depth data during the training procedure. Second, we propose a joint learning scheme realized by combining supervised and self-supervised learning. The weakness of each learning is compensated, thus leading to more accurate depth estimation. Third, we propose a non-local fusion block, which can further retain the global information encoded by vision transformer when reconstructing the depths. With the proposed methods, we successfully apply the transformer to 360 depth estimations, to the best of our knowledge, which has not been tried before. On several benchmarks, our approach achieves significant improvements over previous works and establishes a state of the art.

Robust photon-efficient imaging using a pixel-wise residual shrinkage network.

Single-photon light detection and ranging (LiDAR) has been widely applied to 3D imaging in challenging scenarios. However, limited signal photon counts and high noises in the collected data have posed great challenges for predicting the depth image precisely. In this paper, we propose a pixel-wise residual shrinkage network for photon-efficient imaging from high-noise data, which adaptively generates the optimal thresholds for each pixel and denoises the intermediate features by soft thresholding. Besides, redefining the optimization target as pixel-wise classification provides a sharp advantage in producing confident and accurate depth estimation when compared with existing research. Comprehensive experiments conducted on both simulated and real-world datasets demonstrate that the proposed model outperforms the state-of-the-arts and maintains robust imaging performance under different signal-to-noise ratios including the extreme case of 1:100.

Bathymetric mapping and estimation of water storage in a shallow lake using a remote sensing inversion method based on machine learning

ABSTRACT Accurate lake depth mapping and estimation of changes in water level and water storage are fundamental significance for understanding the lake water resources on the Tibetan Plateau. In this study, combined with satellite images and bathymetric data, we comprehensively evaluate the accuracy of a multi-factor combined linear regression model (MLR) and machine learning models, create a depth distribution map and compare it with the spatial interpolation, and estimate the change of water level and water storage based on the inverted depth. The results indicated that the precision of the random forest (RF) was the highest with a coefficient of determination (R 2) value (0.9311) and mean absolute error (MAE) values (1.13 m) in the test dataset and had high reliability in the overall depth distribution. The water level increased by 9.36 m at a rate of 0.47 m/y, and the water storage increased by 1.811 km3 from 1998 to 2018 based on inversion depth. The water level change was consistent with that of the Shuttle Radar Topography Mission (SRTM) method. Our work shows that this method may be employed to study the water depth distribution and its changes by combining with bathymetric data and satellite imagery in shallow lakes.