Low-textured Regions Research Articles

SIM-MultiDepth: Self-Supervised Indoor Monocular Multi-Frame Depth Estimation Based on Texture-Aware Masking

Self-supervised monocular depth estimation methods have become the focus of research since ground truth data are not required. Current single-image-based works only leverage appearance-based features, thus achieving a limited performance. Deep learning based multiview stereo works facilitate the research on multi-frame depth estimation methods. Some multi-frame methods build cost volumes and take multiple frames as inputs at the time of test to fully utilize geometric cues between adjacent frames. Nevertheless, low-textured regions, which are dominant in indoor scenes, tend to cause unreliable depth hypotheses in the cost volume. Few self-supervised multi-frame methods have been used to conduct research on the issue of low-texture areas in indoor scenes. To handle this issue, we propose SIM-MultiDepth, a self-supervised indoor monocular multi-frame depth estimation framework. A self-supervised single-frame depth estimation network is introduced to learn the relative poses and supervise the multi-frame depth learning. A texture-aware depth consistency loss is designed considering the calculation of the patch-based photometric loss. Only the areas where multi-frame depth prediction is considered unreliable in low-texture regions are supervised by the single-frame network. This approach helps improve the depth estimation accuracy. The experimental results on the NYU Depth V2 dataset validate the effectiveness of SIM-MultiDepth. The zero-shot generalization studies on the 7-Scenes and Campus Indoor datasets aid in the analysis of the application characteristics of SIM-MultiDepth.

Multi-view depth estimation based on multi-feature aggregation for 3D reconstruction

Robust 3D reconstruction depends critically on the accuracy of depth maps, and multi-view depth estimate is greatly aided by Multi-View Stereo (MVS). Nevertheless, challenges still exist, particularly in accurately estimating depth in regions with low texture and occlusion edges. We introduce MFA-MVSNet, a novel Multi-Feature Aggregation Multi-View Stereo Network that aggregates diverse features in order to address these issues. The Local Feature Extractor (LFE) is incorporated into MFA-MVSNet to adaptively capture local features across various receptive fields. It also incorporates a Multi-feature Fusion Module (MFM) to combine local features with global features extracted by the Global Feature Extractor (GFE). We utilize neural edge information in a CNN-based Depth Refinement Module (DRM) to iteratively filter the depth map to address depth errors in low-textured regions and occlusion edges. To further improve network robustness and training efficiency, we also introduce a Region Perception Loss (RPL) to lessen the impact of both easily matched and mismatched areas. On both the DTU and BlendedMVS datasets, experimental results demonstrate that MFA-MVSNet outperforms recent advanced methods in terms of depth map quality. Additionally, we incorporate MFA-MVSNet into a multi-view 3D reconstruction pipeline, utilizing its depth maps to enhance downstream tasks such as point cloud reconstruction and neural radiance fields (NeRF), leading to improved 3D reconstruction performance.

Efficiency-Accuracy Trade-Off in Light Field Estimation with Cost Volume Construction and Aggregation.

The Rich spatial and angular information in light field images enables accurate depth estimation, which is a crucial aspect of environmental perception. However, the abundance of light field information also leads to high computational costs and memory pressure. Typically, selectively pruning some light field information can significantly improve computational efficiency but at the expense of reduced depth estimation accuracy in the pruned model, especially in low-texture regions and occluded areas where angular diversity is reduced. In this study, we propose a lightweight disparity estimation model that balances speed and accuracy and enhances depth estimation accuracy in textureless regions. We combined cost matching methods based on absolute difference and correlation to construct cost volumes, improving both accuracy and robustness. Additionally, we developed a multi-scale disparity cost fusion architecture, employing 3D convolutions and a UNet-like structure to handle matching costs at different depth scales. This method effectively integrates information across scales, utilizing the UNet structure for efficient fusion and completion of cost volumes, thus yielding more precise depth maps. Extensive testing shows that our method achieves computational efficiency on par with the most efficient existing methods, yet with double the accuracy. Moreover, our approach achieves comparable accuracy to the current highest-accuracy methods but with an order of magnitude improvement in computational performance.

CFDepthNet: Monocular Depth Estimation Introducing Coordinate Attention and Texture Features

Handling the depth estimation of low-texture regions using photometric error loss is a challenge due to the difficulty of achieving convergence due to the presence of multiple local minima for pixels in low-texture regions (or even no-texture regions). In this paper, based on the photometric loss, we also introduce texture feature metric loss as a constraint and combine the coordinate attention mechanism to improve the depth map's texture quality and edge detail. This paper uses a simple yet compact network structure, a unique loss function, and a relatively flexible embedded attention module, which is more effective and easier to arrange in robotic platforms with weak arithmetic power. The tests show that our network structure not only shows high quality and state-of-the-art results on the KITTI dataset, but the same training results also perform well on the cityscapes and Make3D datasets.

F[formula omitted]Depth: Self-supervised indoor monocular depth estimation via optical flow consistency and feature map synthesis

Self-supervised monocular depth estimation methods have been increasingly given much attention due to the benefit of not requiring large, labelled datasets. Such self-supervised methods require high-quality salient features and consequently suffer from severe performance drop for indoor scenes, where low-textured regions dominant in the scenes are almost indiscriminative. To address the issue, we propose a self-supervised indoor monocular depth estimation framework called F2Depth. A self-supervised optical flow estimation network is introduced to supervise depth learning. To improve optical flow estimation performance in low-textured areas, only some patches of points with more discriminative features are adopted for finetuning based on our well-designed patch-based photometric loss. The finetuned optical flow estimation network generates high-accuracy optical flow as a supervisory signal for depth estimation. Correspondingly, an optical flow consistency loss is designed. Multi-scale feature maps produced by finetuned optical flow estimation network perform warping to compute feature map synthesis loss as another supervisory signal for depth learning. Experimental results on the NYU Depth V2 dataset demonstrate the effectiveness of the framework and our proposed losses. To evaluate the generalization ability of our F2Depth, we collect a Campus Indoor depth dataset composed of approximately 1500 points selected from 99 images in 18 scenes. Zero-shot generalization experiments on 7-Scenes dataset and Campus Indoor achieve δ1 accuracy of 75.8% and 76.0% respectively. The accuracy results show that our model can generalize well to monocular images captured in unknown indoor scenes.

End-to-end information fusion method for transformer-based stereo matching

In stereo matching, the application of transformers can overcome the limitations of disparity range and capture long-range matching information. However, the lack of cross-epipolar context information often leads to numerous mismatches, especially in low-texture regions. An end-to-end information fusion stereo matching method is proposed to address this issue. In the proposed method, a feature extraction method that combines dense connections and a residual block is proposed. Global and local semantic information can be effectively fused by incorporating dense connections among multiscale feature maps. Additionally, the inclusion of a residual block helps extract more representative feature maps. The idea of criss-cross attention is introduced in the transformer implicit matching process. Criss-cross attention enables the capture of cross-epipolar context information by combining horizontal and vertical attention mechanisms. This method improves the matching accuracy from the perspective of multi-path information fusion. According to the matching results, the disparity regression layer and the context adjustment layer are used to generate the initial and final disparity maps, respectively. The proposed method is evaluated on the Scene Flow, KITTI 2012, and Middlebury 2014 datasets. Experimental results indicate that the proposed method effectively enhances matching accuracy. Moreover, the proposed method exhibits strong generalization ability, allowing for direct application to synthetic, real outdoor, and real indoor scene images.

A study of stereo matching algorithm on low texture and depth discontinuity regions

<p>This article studies the performance of the proposed stereo matchingalgorithm on complex regions. These regions are areas with very limitedinformation for the matching process which are low texture, and depthdiscontinuity regions. In this study, each algorithm uses different matchingcost computation (MCC) techniques, but for cost aggregation (CA), disparityoptimization (DO) and disparity refinement (DR), the technique remains thesame. The MCC areabsolute difference(AD), the combination ofabsolutedifference and gradient matching(AD+GM) andcensus transform(CT).Then, for CA, DO and DR, they areminimum spanning tree(MST),winnertake all(WTA) andbilateral filter(BF), respectively. The results are presentedand discussed in this article. Hence, thru this study the robust method can beestimated at the MCC stage.</p>

Scene-aware refinement network for unsupervised monocular depth estimation in ultra-low altitude oblique photography of UAV

Using depth estimation joint target detection networks to locate targets in the UAV field of view is a novel application in the depth estimation research field. The presence of more depth variations and low-texture regions in the ultra-low altitude oblique photographic images make them trickier to train for an excellent depth estimation network compared to autonomous driving scenarios. This presents a challenge in achieving optimal training. This study investigates the problem of unsupervised monocular depth estimation for ultra-low altitude oblique photography images. It aims to make subsequent advanced vision tasks better benefit from excellent depth estimation results in terms of overcoming complex scenes. The lack of effective back-projection directionality in training using adjacent frames is attributed to the extensive low-textured areas contained in the training data for complex ultra-low altitude oblique photography. We propose a self-supervised scene-aware refinement learning architecture from the perspective of enhancing feature perception to deal with such problems. The architecture consists of a multi-resolution feature fusion depth network and a perceptual refinement network (PRNet), together with a pose network to enhance regional differences in complex environments from a refined feature context perspective to obtain higher quality depth maps. We rethink the problem of depth information recovery and design the edge information aggregation (EIA) module, which is configured in the decoder section to refine the local region depth detail representation. We design several loss terms to constrain the training of the network in order to improve the quality of depth estimation. Our method is compared with six state-of-the-art self-supervised monocular depth estimation methods on three datasets (UAVid 2020, WildUAV, UAV ula). The experimental results demonstrate that our model achieves the best performance in most scenarios. The code and the private dataset (UAV ula) can be publicly available at https://github.com/takisu0916/MRFEDepth.

TOF-assisted binocular vision accuracy improvement method for underwater fish size inspection.

To address problems such as the lack of accuracy in acquiring depth maps for dynamic fish 3D measurements by usual binocular vision or a time-of-flight (TOF) depth camera, a TOF-assisted binocular vision depth acquisition algorithm is used to obtain high-quality depth maps. The TOF depth energy function is designed to guide the binocular stereo matching process, which improves the correct matching rate of binocular matching in low-texture regions; the TOF and binocular stereo matching confidence weighting functions are designed to achieve the fusion of the two at pixel level to improve the matching quality of fish in the occluded overlapping regions. The experimental results show that the TOF-assisted binocular vision system improves the accuracy of fish size measurement compared to single binocular vision while reducing the measurement error when the fish body has a significant inclination along the depth axis.

Fast and robust for texture-less feature registration via adaptive heterogeneous kernels

Feature registration is a core problem in computer vision and machine learning techniques. In recent years, significant progress has been made in learning-based feature registration algorithms. However, such methods usually capture image-level features and strive to capture contour information, inadvertently neglecting the exploration of low-texture regions, while the complex model structure brings more parameters. Therefore, it remains a challenging task to effectively address low-texture feature registration under limited sample conditions. In this paper, we propose a fast and robust feature registration algorithm to achieve fast feature registration in the low-texture case. Specifically, we first construct a two-step learning framework and use a faster adaptive heterogeneous filter (AdaH-Filter) to guide coarse-grained and fine-grained feature extraction. Then, we propose registration feature refiners (ReFR) to achieve the fusion of two-stage features in a low-texture environment while guaranteeing the speed of the detector. Finally, to further improve the network performance, a Warp Extractor is proposed to improve the speed and portability of model inference. This strategy allows the network to perform different levels of fine extraction adaptively according to the registration requirements. Moreover, the proposed method is trained in an end-to-end manner, guided by cosine loss, without any additional supervision. Extensive experimental results on several challenging and representative feature registration datasets in the industry show that the proposed method outperforms the existing state-of-the-art methods. More importantly, it has a faster inference speed under different sensor modalities.

Multistage Pixel-Visibility Learning With Cost Regularization for Multiview Stereo

Multiple-view stereo has potential applications in robotic operations and autonomous driving (unstructured environment construction, visual servo). With assisted depth information, inertial navigation systems can achieve precise navigation. It is, especially suitable for GPS failures in complex environments. Accurate depth estimation is a challenge in low-textured or occluded regions. To alleviate the inference of incorrect depth, a multi-stage pixel-visibility learning-based stereo network is presented in this paper. Its improvements are as follows: 1) a new content-adaptive cost volume aggregation mechanism based on neighboring pixel-wise visibility is designed to effectively produce more accurate and smoother depth map predictions in the object boundary. 2) global convolution block and boundary refinement block are developed to regularize its cost volume, they can learn the inherent constraints of feature matching correspondence and effectively mitigate the depth estimation uncertainty in low-textured regions. 3) a new loss function is designed to measure the uncertainty of predicted probability distribution and enhance the reliability of depth map inference. Experimental results on the indoor DTU datasets and the outdoor Tanks & Temples datasets indicate that our method can achieve superior performance and has a powerful generalization ability, which is comparable to state-of-the-art works. Note to Practitioners—Multiple-view stereo (MVS) can estimate dense 3D representations of scenes, which is widely used in autonomous driving, robotic navigation, virtual reality (VR), and augmented reality (AR). Aiming at the problem of incorrect depth inference in low-textured or occluded regions, this work proposes a novel multi-stage depth prediction method based on neighboring pixel-wise visibility. Our method cannot only achieve accurate depth estimation for robot perception but also make no concession to real-time performance. It is clear that the proposed method has good potential in 3D reconstruction, robotic navigation, and VR/AR fields to provide accurate depth estimation in real-time with limited memory consumption.

Semi-Global Stereo Matching Algorithm Based on Multi-Scale Information Fusion

Semi-global matching (SGM) has been widely used in binocular vision. In spite of its good efficiency, SGM still has difficulties in dealing with low-texture regions. In this paper, an SGM algorithm based on multi-scale information fusion (MSIF), named SGM-MSIF, is proposed by combining multi-path cost aggregation and cross-scale cost aggregation (CSCA). Firstly, the stereo pairs at different scales are obtained by Gaussian pyramid down-sampling. The initial matching cost volumes at different scales are computed by combining census transform and color information. Then, the multi-path cost aggregation in SGM is introduced into the cost aggregation at each scale and the aggregated cost volumes are fused by CSCA. Thirdly, the disparity map is optimized by internal left-right consistency check and median filter. Finally, experiments are conducted on Middlebury datasets to evaluate the proposed algorithm. Experimental results show that the average error matching rate (EMR) of the proposed SGM-MSIF algorithm reduced by 1.96% compared with SGM. Compared with classical cross-scale stereo matching algorithm, the average EMR of SGM-MSIF algorithm reduced by 0.92%, while the processing efficiency increased by 58.7%. In terms of overall performance, the proposed algorithm outperforms the classic SGM and CSCA algorithms. It can achieve high matching accuracy and high processing efficiency for binocular vision applications, especially for those with low-texture regions.

PlaneFusion: Real-Time Indoor Scene Reconstruction With Planar Prior

Real-time dense SLAM techniques aim to reconstruct the dense three-dimensional geometry of a scene in real time with an RGB or RGB-D sensor. An indoor scene is an important type of working environment for these techniques. The planar prior can be used in this scenario to improve the reconstruction quality, especially for large low-texture regions that commonly occur in an indoor scene. This article fully explores the planar prior in a dense SLAM pipeline. First, we propose a novel plane detection and segmentation method that runs at 200 Hz on a modern graphics processing unit. Our algorithm for constructing global plane constraints is very efficient; hence, we use it in the process of each input frame for the camera pose estimation while maintaining the real-time performance. Second, we propose herein a plane-based map representation that greatly reduces the memory footprint of plane regions while keeping the geometric details on planes. The experiments reveal that our system yields superior reconstruction results with planar information running at more than 30 fps. Aside from speed and storage improvements, our technique also handles the low-texture problem in plane regions.

Coarse TRVO: A Robust Visual Odometry with Detector-Free Local Feature

The visual SLAM system requires precise localization. To obtain consistent feature matching results, visual features acquired by neural networks are being increasingly used to replace traditional manual features in situations with weak texture, motion blur, or repeated patterns. However, to improve the level of accuracy, most deep learning enhanced SLAM systems, which have a decreased efficiency. In this paper, we propose Coarse TRVO, a visual odometry system that uses deep learning for feature matching. The deep learning network uses a CNN and transformer structures to provide dense high-quality end-to-end matches for a pair of images, even under indistinctive settings with low-texture regions or repeating patterns occupying the majority of the field of view. Meanwhile, we made the proposed model compatible with NVIDIA TensorRT runtime to boost the performance of the algorithm. After obtaining the matching point pairs, the camera pose is solved in an optimized way by minimizing the re-projection error of the feature points. Experiments based on multiple data sets and real environments show that Coarse TRVO achieves a higher robustness and relative positioning accuracy in comparison with the current mainstream visual SLAM system.

MDSNet: self-supervised monocular depth estimation for video sequences using self-attention and threshold mask

Solving the problem that features prediction of self-supervised monocular depth estimation is still ambiguous in low-texture regions and boundaries. We proposed an innovative self-supervised monocular depth estimation method, monocular depth self-supervised network, which integrates three effective strategies to construct an innovative self-supervised monocular depth estimation framework: (1) the attention mechanism and feature fusion module are adopted to enhance the semantic and spatial information of feature images, (2) the threshold segmentation mask is utilized to solve object motion and low-texture regions to increase image details, and (3) the residual pose module and deep reconstruction loss are used to enhance the feature extraction capability of the model to improve the accuracy of depth and pose estimation. Comprehensive experiments and visual analysis results demonstrate the effectiveness of each component in isolation. Compared to existing self-supervised methods, our model not only achieves outstanding results on KITTI and NYU Depth V2 datasets but also can be suitable to different environments.

Parametric chamfer alignment based on mesh deformation

Alignment for natural images in unconstrained environment is a challenging task. Despite the success for complex deformation, existing feature-based methods may be confused in low-textured regions where features are insufficient, while pixel-based approaches may fail when color changes. In this paper, a parametric chamfer alignment method based on mesh warping model is proposed. Warped positions of mesh vertices are considered as parameters and estimated by optimizing an object function, which measures the chamfer distance of edges and the smoothness of warping. To distinguish the sharpness of pixels, edges are detected through K-means cluster and weights are attached to different levels of edge points. In addition, after the warping model is initialized by feature-based alignment, a growing technique for registering the vertices is presented. Experiment shows that the proposed method outperforms some state-of-the-arts on real data and can be applied in stitching ceramic sanitary ware images.

Development of depth map from stereo images using sum of absolute differences and edge filters

This article proposes a framework for the depth map reconstruction using stereo images. Fundamentally, this map provides an important information which commonly used in essential applications such as autonomous vehicle navigation, drone’s navigation and 3D surface reconstruction. To develop an accurate depth map, the framework must be robust against the challenging regions of low texture, plain color and repetitive pattern on the input stereo image. The development of this map requires several stages which starts with matching cost calculation, cost aggregation, optimization and refinement stage. Hence, this work develops a framework with sum of absolute difference (SAD) and the combination of two edge preserving filters to increase the robustness against the challenging regions. The SAD convolves using block matching technique to increase the efficiency of matching process on the low texture and plain color regions. Moreover, two edge preserving filters will increase the accuracy on the repetitive pattern region. The results show that the proposed method is accurate and capable to work with the challenging regions. The results are provided by the Middlebury standard dataset. The framework is also efficiently and can be applied on the 3D surface reconstruction. Moreover, this work is greatly competitive with previously available methods.

A new function of stereo matching algorithm based on hybrid convolutional neural network

This paper proposes a new hybrid method between the learning-based and handcrafted methods for a stereo matching algorithm. The main purpose of the stereo matching algorithm is to produce a disparity map. This map is essential for many applications, including three-dimensional (3D) reconstruction. The raw disparity map computed by a convolutional neural network (CNN) is still prone to errors in the low texture region. The algorithm is set to improve the matching cost computation stage with hybrid CNN-based combined with truncated directional intensity computation. The difference in truncated directional intensity value is employed to decrease radiometric errors. The proposed method’s raw matching cost went through the cost aggregation step using the bilateral filter (BF) to improve accuracy. The winner-take-all (WTA) optimization uses the aggregated cost volume to produce an initial disparity map. Finally, a series of refinement processes enhance the initial disparity map for a more accurate final disparity map. This paper verified the performance of the algorithm using the Middlebury online stereo benchmarking system. The proposed algorithm achieves the objective of generating a more accurate and smooth disparity map with different depths at low texture regions through better matching cost quality.

Multi-View Stereo Using Graph Cuts-Based Depth Refinement

Multi-View Stereo (MVS) methods tackle the ill-posed inverse problem of recovering an object&#x0027;s 3D structure from its multi-view calibrated images. High computational cost restricts most MVS methods from using global information for depth estimation. We present a depth map-based MVS method that uses global information to estimate the depths of all pixels in an image simultaneously. To this end, we transform the depth refinement problem into computing max-flow/min-cut on a 3D grid graph with offset vertices. The <inline-formula><tex-math notation="LaTeX">$s{-}t$</tex-math></inline-formula> min-cut of this graph corresponds to the minimization of an energy functional consisting of photo-consistency and smoothness terms. Experimental results on indoor and outdoor datasets validate the efficacy of our method, especially on models with low textured regions where global information is necessary to infer the correct depth.

SG-SRNs: Superpixel-Guided Scene Representation Networks

Recently, Scene Representation Networks (SRNs) have attracted increasing attention in computer vision, due to their continuous and light-weight scene representation ability. However, SRNs generally perform poorly on low-texture image regions. Addressing this problem, we propose superpixel-guided scene representation networks in this paper, called SG-SRNs, consisting of a backbone module (SRNs), a superpixel segmentation module, and a superpixel regularization module. In the proposed method, except for the novel view synthesis task, the task of representation-aware superpixel segmentation mask generation is realized by the proposed superpixel segmentation module. Then, the superpixel regularization module utilizes the superpixel segmentation mask to guide the backbone to be learned in a locally smooth way, and optimizes the scene representations of the local regions to indirectly alleviate the structure distortion of low-texture regions in a self-supervised manner. Extensive experimental results on both our constructed datasets and the public Synthetic-NeRF dataset demonstrated that the proposed SG-SRNs achieved a significantly better 3D structure representing performance.