Depth Estimation Network Research Articles

Enhanced light field depth estimation through occlusion refinement and feature fusion

Light field depth estimation is crucial for various applications, but current algorithms often falter when dealing with complex textures and edges. To address this, we propose a light field depth estimation network based on multi-scale fusion and channel attention (LFMCNet). It incorporates a convolutional multi-scale fusion module to enhance feature extraction and utilizes a channel attention mechanism to refine depth map accuracy. Additionally, LFMCNet integrates the Transformer Feature Fusion Module (TFFM) and Channel Attention-Based Perspective Fusion (CAPF) module for improved occlusion refinement, effectively handling challenges in occluded regions. Testing on the 4D HCI and real-world datasets demonstrates that LFMCNet significantly reduces the Bad Pixel (BP) rate and Mean Square Error (MSE).

Analysis of modeling nasal narrow space based on visual slam

Abstract. Now, surgeries are becoming more stable, safe, efficient and low-cost. During the surgical treatment of nasal diseases, surgical robotic robotics can help operate accurately and reduce the discomfort after surgery. However, due to the internal space of the nasal cavity being relatively narrow, it is difficult for the nasal surgical robot to contain multiple vision sensors and the monocular camera could not get information about the depth of the 3D objects in the scene, so the existing surgical robots cannot accomplish the three-dimensional modeling about the internal space of nasal cavity well. In practice, doctors still have to analyze pictures from the robotic, which may decrease the efficiency of the surgery and increase the risk to patients. This article designed a SLAM algorithm framework based on a depth estimation network, it can simulate the internal structure of the nasal cavity more accurately through pictures, which come from monocular endoscopic on surgical robotic. The insights gained in this study verify that the method of image segmentation can also make the depth representation of the nasal internal space more accurate and this method may help robots realize their self-position in the narrow area of the nasal cavity, which lays the foundations for the development of fully autonomous surgical robots.

Edge enhancement and feature modulation based network for light field depth estimation

Estimating depth from light field images is a critical issue in light field applications. While learning-based methods have made significant strides in light field depth estimation, achieving high accuracy and speed simultaneously remains a major challenge. This paper proposes a light field depth estimation network based on edge enhancement and feature modulation, which significantly improves depth estimation results by emphasizing inter-view correlations while preserving image edge features. Specifically, to prioritize edge details, we introduce an Edge-Enhanced Cost Constructor (EECC) that integrates edge information with existing cost constructors to improve depth estimation performance in complex areas. Furthermore, most light field depth estimation networks utilize only sub-aperture images (SAIs) without considering the inherent angular information in macro-pixel image (MacPI). To address this limitation, we propose the MacPI-Guided Feature Modulation (MGFM) module, which leverages angular information between different views in MacPI to modulate features at each view. Experimental results show that our method not only performs excellently on synthetic datasets but also demonstrates outstanding generalization on real-world datasets, achieving a better balance between accuracy and computation speed.

SPDepth: Enhancing Self-Supervised Indoor Monocular Depth Estimation via Self-Propagation

Due to the existence of low-textured areas in indoor scenes, some self-supervised depth estimation methods have specifically designed sparse photometric consistency losses and geometry-based losses. However, some of the loss terms cannot supervise all the pixels, which limits the performance of these methods. Some approaches introduce an additional optical flow network to provide dense correspondences supervision, but overload the loss function. In this paper, we propose to perform depth self-propagation based on feature self-similarities, where high-accuracy depths are propagated from supervised pixels to unsupervised ones. The enhanced self-supervised indoor monocular depth estimation network is called SPDepth. Since depth self-similarities are significant in a local range, a local window self-attention module is embedded at the end of the network to propagate depths in a window. The depth of a pixel is weighted using the feature correlation scores with other pixels in the same window. The effectiveness of self-propagation mechanism is demonstrated in the experiments on the NYU Depth V2 dataset. The root-mean-squared error of SPDepth is 0.585 and the δ1 accuracy is 77.6%. Zero-shot generalization studies are also conducted on the 7-Scenes dataset and provide a more comprehensive analysis about the application characteristics of SPDepth.

Lightweight monocular depth estimation using a fusion-improved transformer

The existing deep estimation networks often overlook the issue of computational efficiency while pursuing high accuracy. This paper proposes a lightweight self-supervised network that combines convolutional neural networks (CNN) and Transformers as the feature extraction and encoding layers for images, enabling the network to capture both local geometric and global semantic features for depth estimation. First, depth-separable convolution is used to construct a dilated convolution residual module based on a shallow network to improve the shallow CNN feature extraction receptive field. In the transformer, a multidepth separable convolution head transposed attention module is proposed to reduce the computational burden of spatial self-attention. In the feedforward network, a two-step gating mechanism is proposed to improve the nonlinear representation ability of the feedforward network. Finally, the CNN and transformer are integrated to implement a depth estimation network with a local-global context interaction function. Compared with other lightweight models, this model has fewer model parameters and higher estimation accuracy. It also has better generalizability for different outdoor datasets. Additionally, the inference speed can reach 87 FPS, achieving better real-time performance and accounting for both inference speed and estimation accuracy.

LDA-Mono: A lightweight dual aggregation network for self-supervised monocular depth estimation

Monocular depth estimation plays a crucial role in various computer vision and robotics applications, particularly in self-supervised methods that do not require ground-truth depth maps. The scene structure and local details are important cues for accurate depth estimation. Most recent self-supervised monocular depth estimation studies have used convolutional neural networks or networks fused with Transformers for inference. However, they primarily model global contextual relationships and lack accurate scene structure perception and proper local detail processing. To address these problems, we propose LDA-Mono, a lightweight dual-aggregation self-supervised depth estimation network. First, in response to edge blurring, we propose the Consecutive Adaptive Dilated Convolution (CADC) module. It utilises multi-layer dilated convolutions to expand the receptive field of the network and select features adaptively. This enhances the key local details and effectively fuses the multi-scale features. Our proposed Dual Feature Aggregation (DFA) module employs dual self-attention for remote global context modelling. Features are aggregated from the spatial and channel dimensions to improve the perception and representation of complex scene structures. With a well-designed architecture, the proposed model generates depth estimates with clearer details and more accurate results. The experimental results show that LDA-Mono significantly outperforms other state-of-the-art methods in terms of accuracy, with a model size of only 23% of the lightweight Lite-Mono. In addition, extensive experiments verify the effectiveness of the proposed method and demonstrate its excellent generalisability to other datasets.

End-to-end color fringe depth estimation based on a three-branch U-net network

In fringe projection profilometry (FPP), end-to-end depth estimation from fringe patterns for FPP attracts more and more attention from fringe patterns. However, color images provide additional information from the RGB channel for FPP, which has been paid little attention in depth estimation. To this end, in this paper we present for the first time, to the best of our knowledge, an end-to-end network for depth estimation using color composite fringes with better performance. In order to take advantage of the color fringe pattern, a multi-branch structure is designed in this paper, which learns the multi-channel details of the object under test by using three encoders for each RGB channel and introduces an attention module to better capture the complex features and modalities information in the input data. Experiments from simulated and real datasets show that the proposed method with color fringe pattern is effective for depth estimation, and it outperforms other deep learning methods such as UNet, R2Unet, PCTNet, and DNCNN.

Learning deformable hypothesis sampling for patchmatch multi-view stereo in the wild

The learnable PatchMatch formulations have recently made progress in Multi-View Stereo (MVS). However, their performance often degrades under complex wild scenes. In this study, we observe that the degradation is mainly caused by the noisy depth hypothesis sampling during iterations of PatchMatch MVS: (i) Within a single iteration, the method mixes all information of a fixed-shape region in the spatial dimension, which introduces sampling noise, especially in regions with abrupt depth changes; (ii) Between iterations, the depth candidate hypotheses are sampled indiscriminately in the depth dimension, which fails to filter out noise in the sampling range and propagates errors to the next iteration. Accordingly, we propose a deformable sampling learning strategy (DefLearn) guided by matching uncertainty to determine the hypothesis set from two perspectives: (i) Encoding the matching uncertainty within the image allows the search region of sampling to fit the depth distribution in the spatial dimension; (i) Modeling the cross-image matching uncertainty more accurately based on the obtained valid hypotheses to capture scene structure information for fine-grained hypothesis searching in the depth dimension. With DefLearn, we develop a deep probabilistic PatchMatch MVS network (PPMNet) for accurate depth estimation with a high awareness of diverse scene structures. Extensive experiments conducted on three challenging datasets demonstrate that our PPMNet significantly outperforms state-of-the-art methods in terms of accuracy and generalization ability, even in particularly challenging wild scenes. Code is available at https://github.com/Geo-Tell/PPMNet.

Depth estimation from monocular endoscopy using simulation and image transfer approach

Obtaining accurate distance or depth information in endoscopy is crucial for the effective utilization of navigation systems. However, due to space constraints, incorporating depth cameras into endoscopic systems is often impractical. Our goal is to estimate depth images directly from endoscopic images using deep learning. This study presents a three-step methodology for training a depth-estimation network model. Initially, simulated endoscopy images and corresponding depth maps are generated using Unity based on a colon surface model obtained from segmented computed tomography colonography data. Subsequently, a cycle generative adversarial network model is employed to enhance the realism of the simulated endoscopy images. Finally, a deep learning model is trained using the synthesized endoscopy images and depth maps to estimate depths accurately. The performance of the proposed approach is evaluated and compared against prior studies utilizing unsupervised training methods. The results demonstrate the superior precision of the proposed technique in estimating depth images within endoscopy. The proposed depth estimation method holds promise for advancing the field by enabling enhanced navigation, improved lesion marking capabilities, and ultimately leading to better clinical outcomes.

Self-Supervised Monocular Depth Estimation for Endoscopic Imaging.

Endoscopy holds a pivotal role in the early detection and treatment of diverse diseases, with artificial intelligence (AI)-assisted methods increasingly gaining prominence in disease screening. Among them, the depth estimation from endoscopic sequences is crucial for a spectrum of AI-assisted surgical techniques. However, the development of endoscopic depth estimation algorithms presents a formidable challenge due to the unique environmental intricacies and constraints within the dataset. This paper proposes a self-supervised depth estimation network to comprehensively explore the brightness changes in endoscopic images, and fuse different features at multiple levels to achieve an accurate prediction of endoscopic depth. First, a FlowNet is designed to evaluate the brightness changes of adjacent frames by calculating the multi-scale structural similarity. Second, a feature fusion module is presented to capture multi-scale contextual information. Experiments show that the average accuracy of the algorithm is 97.03% in the Stereo Correspondence and Reconstruction of Endoscopic Data (SCARED dataset). Based on the training parameters of the SCARED dataset, the algorithm achieves superior performance on the other two datasets (EndoSLAM and KVASIR dataset), indicating that the algorithm has good generalization performance.

A Transformer‐Based Network for Full Object Pose Estimation with Depth Refinement

In response to increasing demand for robotics manipulation, accurate vision‐based full pose estimation is essential. While convolutional neural networks‐based approaches have been introduced, the quest for higher performance continues, especially for precise robotics manipulation, including in the Agri‐robotics domain. This article proposes an improved transformer‐based pipeline for full pose estimation, incorporating a Depth Refinement Module. Operating solely on monocular images, the architecture features an innovative Lighter Depth Estimation Network using a Feature Pyramid with an up‐sampling method for depth prediction. A Transformer‐based Detection Network with additional prediction heads is employed to directly regress object centers and predict the full poses of the target objects. A novel Depth Refinement Module is then utilized alongside the predicted centers, full poses, and depth patches to refine the accuracy of the estimated poses. The performance of this pipeline is extensively compared with other state‐of‐the‐art methods, and the results are analyzed for fruit picking applications. The results demonstrate that the pipeline improves the accuracy of pose estimation to up to 90.79% compared to other methods available in the literature.

3DGEN: a framework for generating custom-made synthetic 3D datasets for civil structure health monitoring

The availability of high-quality datasets is increasingly critical in the field of computer vision-based civil structural health monitoring, where deep learning approaches have gained prominence. However, the lack of specialized datasets for such tasks poses a significant challenge for training a reliable model. To address this challenge, a framework, 3DGEN, is proposed to swiftly generate realistic synthetic 3D datasets which can be targeted for specific tasks. The framework is based on diverse 3D civil structural models, rendering them from various angles and providing depth information and camera parameters for training neural networks. By employing mathematical methods, such as analytical solutions and/or numerical simulations, deformation of civil engineering structures can be generated, ensuring a reliable representation of their real-world shapes and characteristics in the 3D datasets. For texture generation, a generative 3D texturing method enables users to specify desired textures using plain English sentences. Two successful experiments are conducted to (1) assess the efficiency of generating the 3D datasets using two distinct structures, (2) train a monocular depth estimation network to perform 3D surface reconstruction with the generated dataset. Notably, 3DGEN is not limited to 3D surface reconstruction; it can also be used for training neural networks for various other tasks. The code and dataset are available at: https://github.com/YANDA-SHAO/Beam-Dataset-SE

Stereo matching and 3D reconstruction with NeRF supervision for accurate weight estimation in free-swimming fish

Non-invasive methods of accurate estimation of fish weight are essential for biomass assessment, precision feeding, and effective management in aquaculture. However, the lack of large-scale stereo matching datasets and costly depth sensing equipment in underwater scenarios present significant challenges to accurately measure the size of free-swimming fish. This study introduces a novel NeRF-supervised stereo matching network for precise depth estimation and 3D reconstruction of the fish body. The state-of-the-art neural rendering methods are employed to generate stereo training data from image sequences. In this study, the residual network with deformable convolutions was proposed to extract contextual texture features, significantly enhancing disparity estimation accuracy at the edge of the fish body. Additionally, multiscale spatial features are fused using a feature pyramid network (FPN). To enhance the adaptability of disparity search, we introduce a lightweight Triplet Attention module to capture cross-dimensional dependencies prior to constructing the correlation volume. Experimental results demonstrate the effectiveness of the proposed depth estimation method, achieving an impressive result of 92.35% in the similarity metric, compared to the ground truth. Moreover, we developed a weight estimation model based on bass perimeter using instance segmentation and 3D reconstruction techniques. Notably, a robust correlation (correlation coefficient of 0.98) between fish perimeter and weight is observed, highlighting the potential application of our methods to enhance the accuracy of weight estimation for free-swimming fish.

FeverNet: Enabling accurate and robust remote fever screening

Remote human fever screening via thermal infrared imaging helps reduce the risk of respiratory disease transmission and plays an important role in public health monitoring. However, the accuracy of such systems often falls prey to variations in measurement distance and environment temperature. Most previous methods tend to employ sensors to overcome these variations, which are expensive schemes and have limited performance improvement. To address above problems, this paper presents a novel and robust remote fever screening framework named FeverNet. Specifically, FeverNet introduces depth estimation network and temperature distribution constraints across time periods to reduce the influence of distance variations and environment temperature changes. The fever attention module is thus proposed to enhance feature representation and expand the difference between fever faces and normal ones. In addition, we provide the Extended Thermal Infrared Face dataset (ETIF), which further gives visible images (paired with thermal infrared images) for depth estimation and improve the fever face generated method based on the maximum temperature of the face. Extensive experiments on ETIF demonstrate the advantages of our FeverNet over the state-of-the-art methods.

Redefining Accuracy: Underwater Depth Estimation for Irregular Illumination Scenes.

Acquiring underwater depth maps is essential as they provide indispensable three-dimensional spatial information for visualizing the underwater environment. These depth maps serve various purposes, including underwater navigation, environmental monitoring, and resource exploration. While most of the current depth estimation methods can work well in ideal underwater environments with homogeneous illumination, few consider the risk caused by irregular illumination, which is common in practical underwater environments. On the one hand, underwater environments with low-light conditions can reduce image contrast. The reduction brings challenges to depth estimation models in accurately differentiating among objects. On the other hand, overexposure caused by reflection or artificial illumination can degrade the textures of underwater objects, which is crucial to geometric constraints between frames. To address the above issues, we propose an underwater self-supervised monocular depth estimation network integrating image enhancement and auxiliary depth information. We use the Monte Carlo image enhancement module (MC-IEM) to tackle the inherent uncertainty in low-light underwater images through probabilistic estimation. When pixel values are enhanced, object recognition becomes more accessible, allowing for a more precise acquisition of distance information and thus resulting in more accurate depth estimation. Next, we extract additional geometric features through transfer learning, infusing prior knowledge from a supervised large-scale model into a self-supervised depth estimation network to refine loss functions and a depth network to address the overexposure issue. We conduct experiments with two public datasets, which exhibited superior performance compared to existing approaches in underwater depth estimation.

An Improved MonoDepth2 Algorithm for Vehicle Monocular Depth Estimation

Depth estimation algorithms are of great significance for autonomous vehicle ranging. In order to solve the problems of large errors, fuzzy image generation and detail loss in depth estimation algorithms, this paper proposes an improved algorithm of the MonoDepth2 algorithm. First, the feature extraction network of the original algorithm is changed to the ResNeXt block to improve the running speed and enrich feature information. Secondly, the Convolutional Block Attention Module attention mechanism is added to the depth estimation network to improve the algorithm accuracy and promote detail detection effect. Finally, the Atrous Spatial Pyramid Pooling network structure is added between the encoder and decoder of the depth estimation network to optimize feature fusion at different scales and extract more detailed features. Compared with the original MonoDepth2 algorithm, the improved algorithm reduces the Absolute Relative Error by 0.4 %, the Squared Relative Error by 6.1 %, the Root Mean Square Error by 15 %, and the accuracy(thr=1.25), increased by 0.4 %, The generated depth map recovers deeper image details and a more detailed target contour.

SIM-MultiDepth: Self-Supervised Indoor Monocular Multi-Frame Depth Estimation Based on Texture-Aware Masking

Self-supervised monocular depth estimation methods have become the focus of research since ground truth data are not required. Current single-image-based works only leverage appearance-based features, thus achieving a limited performance. Deep learning based multiview stereo works facilitate the research on multi-frame depth estimation methods. Some multi-frame methods build cost volumes and take multiple frames as inputs at the time of test to fully utilize geometric cues between adjacent frames. Nevertheless, low-textured regions, which are dominant in indoor scenes, tend to cause unreliable depth hypotheses in the cost volume. Few self-supervised multi-frame methods have been used to conduct research on the issue of low-texture areas in indoor scenes. To handle this issue, we propose SIM-MultiDepth, a self-supervised indoor monocular multi-frame depth estimation framework. A self-supervised single-frame depth estimation network is introduced to learn the relative poses and supervise the multi-frame depth learning. A texture-aware depth consistency loss is designed considering the calculation of the patch-based photometric loss. Only the areas where multi-frame depth prediction is considered unreliable in low-texture regions are supervised by the single-frame network. This approach helps improve the depth estimation accuracy. The experimental results on the NYU Depth V2 dataset validate the effectiveness of SIM-MultiDepth. The zero-shot generalization studies on the 7-Scenes and Campus Indoor datasets aid in the analysis of the application characteristics of SIM-MultiDepth.

Self-Supervised Monocular Depth Estimation via Binocular Geometric Correlation Learning

Monocular depth estimation aims to infer a depth map from a single image. Although supervised learning-based methods have achieved remarkable performance, they generally rely on a large amount of labor-intensively annotated data. Self-supervised methods, on the other hand, do not require any annotation of ground-truth depth and have recently attracted increasing attention. In this work, we propose a self-supervised monocular depth estimation network via binocular geometric correlation learning. Specifically, considering the inter-view geometric correlation, a binocular cue prediction module is presented to generate the auxiliary vision cue for the self-supervised learning of monocular depth estimation. Then, to deal with the occlusion in depth estimation, an occlusion interference attenuated constraint is developed to guide the supervision of the network by inferring the occlusion region and producing paired occlusion masks. Experimental results on two popular benchmark datasets have demonstrated that the proposed network obtains competitive results compared to state-of-the-art self-supervised methods and achieves comparable results to some popular supervised methods.

MMAIndoor: Patched MLP and multi-dimensional cross attention based self-supervised indoor depth estimation

Depth estimation can provide auxiliary information for scene perception. Generally, extensive textureless surfaces, such as walls and ceilings, exist in indoor environments, and they share similar scene and semantic content. Overly consistent features of local textureless areas fail to reflect changes in depth information, thus degrading the performance of existing depth estimation methods. In response to this challenge, we propose a special indoor depth estimation method, named as MMAIndoor, which can provide global semantic guidance and shape priors for local textureless depth estimation. The depth estimation network is designed efficiently, encompassing the initial convolutional stage and the latent patched multi-layer perceptron (Pat-MLP) stage. The novel Pat-MLP block utilizes MLP partitioning to globally model depth-local information from the convolutional stage and it incorporate axial shift operations to extract local information from various spatial locations, suppressing the smoothing effect of MLP and improving precise estimation of sharp depth changes or small structures indoors. Further, we build a multi-dimensional cross attention (MCA) module to address the weak correlation of the current residual connections for the global context. This MCA captures global dependencies across multi-dimensions by sequentially executing cross attention on both channels and spatial, and effectively mitigate semantic gaps in residual connections. Sufficient experimental results demonstrate the state-of-the-art performance of MMAIndoor on benchmark datasets including NYUv2, ScanNet, and InteriorNet.

Attention‐optimized 3D segmentation and reconstruction system for sewer pipelines employing multi‐view images

AbstractExisting deep learning‐based defect inspection results on images lack depth information to fully demonstrate the sewer, despite their high accuracy. To address this limitation, a novel attention‐optimized three‐dimensional (3D) segmentation and reconstruction system for sewer pipelines is presented. First, a real‐time sewer segmentation method called AM‐Pipe‐SegNet is developed to inspect defects (i.e., misalignment, obstacle, and fracture) efficiently. Attention mechanisms (AMs) are introduced to improve the performance of segmentation. Second, an attention‐optimized and sparse‐initialized depth estimation network called AM‐Pipe‐DepNet is presented to generate depth maps from multi‐view images. Third, a 2D‐to‐3D mapping algorithm is proposed to remove noise and transform the sewer segmentation results into 3D spaces. Comparison experiments reveal that incorporating AMs into the network significantly enhances pipe segmentation and 3D reconstruction performance. Finally, two digital replicas of real sewer pipes are built based on photos taken by probes, providing valuable insights for sewer maintenance.