Depth Estimation Results Research Articles

Optimizing Underwater Image Restoration and Depth Estimation with Light Field Images

Methods based on light field information have shown promising results in depth estimation and underwater image restoration. However, improvements are still needed in terms of depth estimation accuracy and image restoration quality. Previous work on underwater image restoration employed an image formation model (IFM) that overlooked the effects of light attenuation and scattering coefficients in underwater environments, leading to unavoidable color deviation and distortion in the restored images. Additionally, the high blurriness and associated distortions in underwater images make depth information extraction and estimation very challenging. In this paper, we refine the light propagation model and propose a method to estimate the attenuation and backscattering coefficients of the underwater IFM. We simplify these coefficients into distance-related functions and design a relationship between distance and the darkest channel to estimate the water coefficients, effectively suppressing color deviation and distortion in the restoration results. Furthermore, to increase the accuracy of depth estimation, we propose using blur cues to construct a cost for refocusing in the depth direction, reducing the impact of high signal-to-noise ratio environments on depth information extraction, and effectively enhancing the accuracy and robustness of depth estimation. Finally, experimental comparisons show that our method achieves more accurate depth estimation and image restoration closer to real scenes compared to state-of-the-art methods.

Extended depth of field for Fresnel zone aperture camera via fast passive depth estimation.

The lensless camera with incoherent illumination has gained significant research interest for its thin and flexible structure. However, it faces challenges in resolving scenes with a wide depth of field (DoF) due to its depth-dependent point spread function (PSF). In this paper, we present a single-shot method for extending the DoF in Fresnel zone aperture (FZA) cameras at visible wavelengths through passive depth estimation. The improved ternary search method is utilized to determine the depth of targets rapidly by evaluating the sharpness of the back propagation reconstruction. Based on the depth estimation results, a set of reconstructed images focused on targets at varying depths are derived from the encoded image. After that, the DoF is extended through focus stacking. The experimental results demonstrate an 8-fold increase compared with the calibrated DoF at 130 mm depth. Moreover, our depth estimation method is five times faster than the traversal method, while maintaining the same level of accuracy. The proposed method facilitates the development of lensless imaging in practical applications such as photography, microscopy, and surveillance.

Moving harmonic source depth estimation by MDFMD-v in a weakly range-depend waveguide

To estimate the depth of a non-cooperative moving sound source in a range-depend waveguide, a passive depth estimation method combining depth-polarity search for a harmonic moving source is proposed in this paper. Firstly, the method obtains each mode parameters at the source and horizontal linear array (HLA) locations by utilizing the modal depth function constrained modal Doppler velocity estimation (MDFMD-v) and matrix pencil (MP) methods, respectively. Subsequently, mode amplitudes are calculated by compressive sensing. To determine the polarity of each mode amplitude, a depth-polarity search method is given in this paper. Finally, a depth ambiguity function with mode amplitude polarity is used to obtain the depth estimation results. Simulation results verify the effectiveness of the method.

Joint Learning of Image Deblurring and Depth Estimation Through Adversarial Multi-Task Network

Self-supervised monocular depth estimation methods have achieved remarkable results on natural clear images. However, it is still a serious challenge to directly recover depth information from blurred images caused by long-time exposure while camera fast moving. To address this issue, we propose a unified framework for simultaneous deblurring and depth estimation (SDDE), which has higher coupling performance and flexibility compared with the simple concatenation strategy of deblurring model and depth estimation model. This framework mainly benefits from three features: (1) a novel Task-aware Fusion Module (TFM) to adaptively select the most relevant intermediate shared features for the dual decoder network by aggregating multi-scale features, (2) a unique Spatial Interaction Module (SIM) to learn higher-order representation in the encoder stage to better describe complex boundaries of different classes in high-dimensional space, and focuses on the task-related region by modeling the pairwise spatial correlation of the holistic tensor, (3) a Priors-Based Composite Regularization term to jointly optimize the shared encoder-dual decoder network. This work was evaluated on multiple datasets, including: Stereo blur [1], KITTI [2],NYUv2 [3], REDS [4] and our own large-scale stereo blur dataset, resulting in state-of-the-art results for depth estimation and image deblurring, respectively.

Unsupervised Cross-Spectrum Depth Estimation by Visible-Light and Thermal Cameras

Cross-spectrum depth estimation aims to provide a reliable depth map under variant-illumination conditions with a pair of dual-spectrum images. It is valuable for autonomous driving applications when vehicles are equipped with two cameras of different modalities. However, images captured by different-modality cameras can be photometrically quite different, which makes cross-spectrum depth estimation a very challenging problem. Moreover, the shortage of large-scale open-source datasets also retards further research in this field. In this paper, we propose an unsupervised visible light(VIS)-image-guided cross-spectrum (i.e., thermal and visible-light, TIR-VIS in short) depth-estimation framework. The input of the framework consists of a cross-spectrum stereo pair (one VIS image and one thermal image). First, we train a depth-estimation base network using VIS-image stereo pairs. To adapt the trained depth-estimation network to the cross-spectrum images, we propose a multi-scale feature-transfer network to transfer features from the TIR domain to the VIS domain at the feature level. Furthermore, we introduce a mechanism of cross-spectrum depth cycle-consistency to improve the depth estimation result of dual-spectrum image pairs. Meanwhile, we release to society a large cross-spectrum dataset with visible-light and thermal stereo images captured in different scenes. The experiment result shows that our method achieves better depth-estimation results than the compared existing methods. Our code and dataset are available on https://github.com/whitecrow1027/CrossSP_Depth.

Determining optimal cochlear implant electrode array with OTOPLAN

Background To achieve better speech performance following cochlear implantation (CI), measuring the patient’s cochlear duct length (CDL) and determining the appropriate length of the CI array are important. Objective To investigate the CDL in CI patients after using the OTOPLAN software preoperatively and compare the results of angular insertion depth (AID) estimation by OTOPLAN and postoperative radiography. Materials and Methods The study included 105 Japanese CI patients with normal cochleae. We measured the CDL using OTOPLAN and the position of the tip channel of the electrode for each selected electrode array, and estimated the AID using the software. Results The mean CDL was 35.1 ± 1.6 mm. Preoperatively, the mean estimated AID was 580.3 ± 57.8°. Postoperative radiography revealed a mean AID of 583.0 ± 56.7°, demonstrating a strong linear correlation between the two measurements (R2 = 0.635). Conclusion and Significance Our findings revealed that CDL varies widely, which is consistent with previous studies. To achieve better speech perception, surgeons should select the appropriate length of CI electrode array based on the individual’s CDL. Preoperative measurement of each CDL by OTOPLAN, which is clinically feasible and comparable to postoperative evaluation, can be used to ensure selection of the appropriate electrode array length.

Research on Fruit Spatial Coordinate Positioning by Combining Improved YOLOv8s and Adaptive Multi-Resolution Model

Automated fruit-picking equipment has the potential to significantly enhance the efficiency of picking. Accurate detection and localization of fruits are particularly crucial in this regard. However, current methods rely on expensive tools such as depth cameras and LiDAR. This study proposes a low-cost method based on monocular images to achieve target detection and depth estimation. To improve the detection accuracy of targets, especially small targets, an advanced YOLOv8s detection algorithm is introduced. This approach utilizes the BiFormer block, an attention mechanism for dynamic query-aware sparsity, as the backbone feature extractor. It also adds a small-target-detection layer in the Neck and employs EIoU Loss as the loss function. Furthermore, a fused depth estimation method is proposed, which incorporates high-resolution, low-resolution, and local high-frequency depth estimation to obtain depth information with both high-frequency details and low-frequency structure. Finally, the spatial 3D coordinates of the fruit are obtained by fusing the planar coordinates and depth information. The experimental results with citrus as the target result in an improved YOLOv8s network mAP of 88.45% and a recognition accuracy of 94.7%. The recognition of citrus in a natural environment was improved by 2.7% compared to the original model. In the detection range of 30 cm~60 cm, the depth-estimation results (MAE, RSME) are 0.53 and 0.53. In the illumination intensity range of 1000 lx to 5000 lx, the average depth estimation results (MAE, RSME) are 0.49 and 0.64. In the simulated fruit-picking scenario, the success rates of grasping at 30 cm and 45 cm were 80.6% and 85.1%, respectively. The method has the advantage of high-resolution depth estimation without constraints of camera parameters and fruit size that monocular geometric and binocular localization do not have, providing a feasible and low-cost localization method for fruit automation equipment.

MODE: Monocular omnidirectional depth estimation via consistent depth fusion

Monocular depth estimation has seen significant progress in recent years, especially in outdoor scenes. However, depth estimation results are not satisfying in omnidirectional images. As compared to perspective images, estimating the depth map from an omnidirectional image captured in the outdoor scene, using neural networks, has two additional challenges: (i) the depth range of outdoor images varies a lot across different scenes, making it difficult for the depth network to predict accurate depth results for training with an indoor dataset, besides the maximum distance in outdoor scenes mostly stays the same as the camera sees the sky, but depth labels in this region are entirely missing in existing datasets; (ii) a standard representation of omnidirectional images introduces spherical distortion, which causes difficulties for the vanilla network to predict accurate relative structural depth details. In this paper, we propose a novel network-MODE by giving special considerations to those challenges and designing a set of flexible modules for improving the performance of omnidirectional depth estimation. First, a consistent depth structure module is proposed to estimate a consistent depth structure map, and the predicted structural map can improve depth details. Second, to suit the characteristics of spherical sampling, we propose a strip convolution fusion module to enhance long-range dependencies. Third, rather than using a single depth decoder branch as in previous methods, we propose a semantics decoder branch to estimate sky regions in the omnidirectional image. The proposed method is validated on three widely used datasets, demonstrating the state-of-the-art performance. Moreover, the effectiveness of each module is shown through an ablation study on real-world datasets. Our code is available at https://github.com/lkku1/MODE.© 2017 Elsevier Inc. All rights reserved.

The HAInich: A multidisciplinary vision data-set for a better understanding of the forest ecosystem

We present a multidisciplinary forest ecosystem 3D perception dataset. The dataset was collected in the Hainich-Dün region in central Germany, which includes two dedicated areas, which are part of the Biodiversity Exploratories - a long term research platform for comparative and experimental biodiversity and ecosystem research. The dataset combines several disciplines, including computer science and robotics, biology, bio-geochemistry, and forestry science. We present results for common 3D perception tasks, including classification, depth estimation, localization, and path planning. We combine the full suite of modern perception sensors, including high-resolution fisheye cameras, 3D dense LiDAR, differential GPS, and an inertial measurement unit, with ecological metadata of the area, including stand age, diameter, exact 3D position, and species. The dataset consists of three hand held measurement series taken from sensors mounted on a UAV during each of three seasons: winter, spring, and early summer. This enables new research opportunities and paves the way for testing forest environment 3D perception tasks and mission set automation for robotics.

Miper-MVS: Multi-scale iterative probability estimation with refinement for efficient multi-view stereo

Multi-view stereo reconstruction aims to construct 3D scenes from multiple 2D images. In recent years, learning-based multi-view stereo methods have achieved significant results in depth estimation for multi-view stereo reconstruction. However, the current popular multi-stage processing method cannot solve the low-efficiency problem satisfactorily owing to the use of 3D convolution and still involves significant amounts of calculation. Therefore, to further balance the efficiency and generalization performance, this study proposed a multi-scale iterative probability estimation with refinement, which is a highly efficient method for multi-view stereo reconstruction. It comprises three main modules: 1) a high-precision probability estimator, dilated-LSTM that encodes the pixel probability distribution of depth in the hidden state, 2) an efficient interactive multi-scale update module that fully integrates multi-scale information and improves parallelism by interacting information between adjacent scales, and 3) a Pi-error Refinement module that converts the depth error between views into a grayscale error map and refines the edges of objects in the depth map. Simultaneously, we introduced a large amount of high-frequency information to ensure the accuracy of the refined edges. Among the most efficient methods (e.g., runtime and memory), the proposed method achieved the best generalization on the Tanks & Temples benchmarks. Additionally, the performance of the Miper-MVS was highly competitive in DTU benchmark. Our code is available at https://github.com/zhz120/Miper-MVS.

Estimation of sedimentary cover thickness by using spectral analysis of gravity anomalies

In this paper, power spectrum analysis has been used for estimating the depth of geological interfaces. The computer code was tested for synthetic models and implemented to gravity survey data to estimate the thickness of sedimentary cover of several Meso-cenozoic basins of Mongolia. The depth estimation results of the basins’ sedimentary cover were in good agreement with other prior geological and geophysical interpretation results of the study areas.

Monocular Depth Estimation Network Based on Swin Transformer

Estimating depth from a single image is challenging because a single 2D image may correspond to many different 3D scenes with the same depth. While most deep learning based depth prediction methods extract depth features using small convolutional kernels with small receptive fields, which results in deformed depth edges and inaccurate depth values of distant objects in the depth estimation results. Aiming at this problem, we propose a depth estimation network based on Swin Transformer and the encoder-decoder structure. We construct the encoder using the Swin Transformer network, which can encode long-range spatial dependency and extract features on various scales and across different channels. The decoder of the proposed network is in charge of fusing the features from the encoder by the operations of interpolation, concatenation, and convolution. Experiments on KITTI and NYUv2 datasets show that our proposed network can get more accurate depth edges and depth values than the state-of-the-art methods.

HI-Net: Boosting Self-Supervised Indoor Depth Estimation via Pose Optimization

Pose estimation plays a critical role in self-supervised monocular depth estimation for indoor scenes, especially those involving complex ego-motion. In this letter, we leverage the two-view geometry constraints into pose estimation to boost the accuracy of pose estimation, which ultimately improves the performance of self-supervised depth estimation. Specifically, we decompose pose estimation into two steps: initial homography estimation and iterative residual refinement. We first introduce a Homography Estimation Module (HEM) to estimate large 3-DoF rotations. Then, we refine the 6-DoF residual pose estimation with an Iterative Residual Refinement Module (IRM). Finally, the supervision signal is generated with the refined pose and used for the training of DepthNet. Experiments on the NYU depth V2 dataset show that our pose estimation approach significantly improves the performance of DepthNet, and the proposed method achieves state-of-the-art depth estimation results. Furthermore, experiments on the ScanNet dataset demonstrate the generalization ability of our method for both pose estimation and depth estimation.

Monocular Accumulated Depth Estimation with Recursive Feature Fusion

Existing depth estimation methods usually use an encoder-decoder structure, but different areas in the image vary in the difficulty and feature demand for depth estimation. This paper proposes a recursive feature fusion monocular depth accumulation estimation method. In the encoder, recursive feature fusion fuses multi-scale features by recursively using gated recurrent units. It extracts features that adapt to the needs of different image areas, replacing cross-layer connections. In the decoder, the depth accumulation estimation decomposes the depth reconstruction process into multiple layers. Different layers predict depth maps of specific granularity and accumulate depth estimation results. Compared with other methods, such as DPT, BTS, and DORN, the proposed method achieved competitive results on two benchmark datasets, KITTI and NYU Depth V2, with Abs Rel reaching 0.058 and 0.107, and RMSE reaching 2.411 and 0.386, respectively.

Recognition and Depth Estimation of Ships Based on Binocular Stereo Vision

To improve the navigation safety of inland river ships and enrich the methods of environmental perception, this paper studies the recognition and depth estimation of inland river ships based on binocular stereo vision (BSV). In the stage of ship recognition, considering the computational pressure brought by the huge network parameters of the classic YOLOv4 model, the MobileNetV1 network was proposed as the feature extraction module of the YOLOv4 model. The results indicate that the mAP value of the MobileNetV1-YOLOv4 model reaches 89.25%, the weight size of the backbone network was only 47.6 M, which greatly reduced the amount of computation while ensuring the recognition accuracy. In the stage of depth estimation, this paper proposes a feature point detection and matching algorithm based on the ORB algorithm at sub-pixel level, that is, firstly, the FSRCNN algorithm was used to perform super-resolution reconstruction of the original image, to further increase the density of image feature points and detection accuracy, which was more conducive to the calculation of the image parallax value. The ships’ depth estimation results indicate that when the distance to the target is about 300 m, the depth estimation error is less than 3%, which meets the depth estimation needs of inland ships. The ship target recognition and depth estimation technology based on BSV proposed in this paper makes up for the shortcomings of the existing environmental perception methods, improves the navigation safety of ships to a certain extent, and greatly promotes the development of intelligent ships in the future.

Multi-Sensor Fusion Self-Supervised Deep Odometry and Depth Estimation

This paper presents a new deep visual-inertial odometry and depth estimation framework for improving the accuracy of depth estimation and ego-motion from image sequences and inertial measurement unit (IMU) raw data. The proposed framework predicts ego-motion and depth with absolute scale in a self-supervised manner. We first capture dense features and solve the pose by deep visual odometry (DVO), and then combine the pose estimation pipeline with deep inertial odometry (DIO) by the extended Kalman filter (EKF) method to produce the sparse depth and pose with absolute scale. We then join deep visual-inertial odometry (DeepVIO) with depth estimation by using sparse depth and the pose from DeepVIO pipeline to align the scale of the depth prediction with the triangulated point cloud and reduce image reconstruction error. Specifically, we use the strengths of learning-based visual-inertial odometry (VIO) and depth estimation to build an end-to-end self-supervised learning architecture. We evaluated the new framework on the KITTI datasets and compared it to the previous techniques. We show that our approach improves results for ego-motion estimation and achieves comparable results for depth estimation, especially in the detail area.

Semantically guided self‐supervised monocular depth estimation

Depth information plays an important role in the vision-related activities of robots and autonomous vehicles. An effective method to obtain 3D scene information is self-supervised monocular depth estimation, which utilizes large and diverse monocular video datasets during the training process without the need for ground-truth data. A novel multi-task learning strategy that uses semantic information to guide the monocular depth estimation method while maintaining self-supervision is proposed. An improved differential direct visual odometer (DDVO) combined with Pose-Net is applied for achieving better pose prediction. Minimum reprojection loss with auto-masking and semantic masking is used to remove the effects of low-texture areas and moving dynamic-class objects within scenes. Concurrently, the semantic masking is introduced into the DDVO pose predictor to filter moving objects and reduce the matching error between monocular sequence frames. In addition, PackNet is employed as the backbone of multi-task learning to further improve the accuracy of deep prediction. The proposed method produces state-of-the-art results for monocular depth estimation on the KITTI Eigen split benchmark, even outperforming supervised methods that have been trained using ground-truth depth.

Single Image Depth Map Estimation for Improving Posture Recognition

Image-based posture recognition is a very challenging problem since it is difficult to acquire rich 3D information from the posture in color image. To address this issue, we present a novel and unified framework for human posture recognition, applying single image depth map estimation from color images. The proposed method includes two stages. The first stage estimates the depth map from the single-color image by an improved Pix2Pix generation module. The generation module is equipped with a hybrid loss function that captures the high-level features and recovers the sharp depth discontinuities, thus improving the depth estimation results. The second stage (the recognition stage) improves the color image-based recognition performance by incorporating the estimated depth map. Thereby, a two-stream CNN architecture that separately processes the color image and its estimated depth image is developed for robust posture recognition. To verify its effectiveness, we first test the proposed method on a novel pose dataset, which contains 13800 samples of paired color-and-depth of 6 subjects with 15 poses. The dataset used in this work is been created and released, is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://media.ritsumei.ac.jp/iipl/database/pose/</uri> . Extensive experiments are also performed on the public OUHANDS hand gesture dataset. Experiments demonstrate that the proposed method achieves superior performance on both human pose and hand gesture recognition tasks.

ADAADepth: Adapting Data Augmentation and Attention for Self-Supervised Monocular Depth Estimation

Self-supervised learning of depth has been a highly studied topic of research as it alleviates the requirement of having ground truth annotations for predicting depth. Depth is learnt as an intermediate solution to the task of view synthesis, utilising warped photometric consistency. Although it gives good results when trained using stereo data, the predicted depth is still sensitive to noise, illumination changes and specular reflections. Also, occlusion can be tackled better by learning depth from a single camera. We propose ADAA, utilising depth augmentation as depth supervision for learning accurate and robust depth. We propose a relational self-attention module that learns rich contextual features and further enhances depth results. We also optimize the auto-masking strategy across all losses by enforcing L1 regularisation over mask. Our novel progressive training strategy first learns depth at a lower resolution and then progresses to the original resolution with slight training. We utilise a ResNet18 encoder, learning features for prediction of both depth and pose. We evaluate our predicted depth on the standard KITTI driving dataset and achieve state-of-the-art results for monocular depth estimation whilst having significantly lower number of trainable parameters in our deep learning framework. We also evaluate our model on Make3D dataset showing better generalization than other methods.

A variational approach for estimation of monocular depth and camera motion in autonomous driving

In this paper, a new direct computational approach to dense 3D reconstruction in autonomous driving is proposed to simultaneously estimate the depth and the camera motion for the motion stereo problem. A traditional Structure from Motion framework is utilized to establish geometric constrains for our variational model. The architecture is mainly composed of the texture constancy constraint, one-order motion smoothness constraint, a second-order depth regularize constraint and a soft constraint. The texture constancy constraint can improve the robustness against illumination changes. One-order motion smoothness constraint can reduce the noise in estimation of dense correspondence. The depth regularize constraint is used to handle inherent ambiguities and guarantee a smooth or piecewise smooth surface, and the soft constraint can provide a dense correspondence as initial estimation of the camera matrix to improve the robustness future. Compared to the traditional dense Structure from Motion approaches and popular stereo approaches, our monocular depth estimation results are more accurate and more robust. Even in contrast to the popular depth from single image networks, our variational approach still has good performance in estimation of monocular depth and camera motion.