Untextured Areas Research Articles

Multiple prior representation learning for self-supervised monocular depth estimation via hybrid transformer

Self-supervised monocular depth estimation aims to infer depth information without relying on labeled data. However, the lack of labeled information poses a significant challenge to the model’s representation, limiting its ability to capture the intricate details of the scene accurately. Prior information can potentially mitigate this issue, enhancing the model’s understanding of scene structure and texture. Nevertheless, solely relying on a single type of prior information often falls short when dealing with complex scenes, necessitating improvements in generalization performance. To address these challenges, we introduce a novel self-supervised monocular depth estimation model that leverages multiple priors to bolster representation capabilities across spatial, context, and semantic dimensions. Specifically, we employ a hybrid transformer and a lightweight pose network to obtain long-range spatial priors in the spatial dimension. Then, the context prior attention is designed to improve generalization, particularly in complex structures or untextured areas. In addition, semantic priors are introduced by leveraging semantic boundary loss, and semantic prior attention is supplemented, further refining the semantic features extracted by the decoder. Experiments on three diverse datasets demonstrate the effectiveness of the proposed model. It integrates multiple priors to comprehensively enhance the representation ability, improving the accuracy and reliability of depth estimation. Codes are available at: https://github.com/MVME-HBUT/MPRLNet.

Stereo Matching Method for Remote Sensing Images Based on Attention and Scale Fusion

With the development of remote sensing satellite technology for Earth observation, remote sensing stereo images have been used for three-dimensional reconstruction in various fields, such as urban planning and construction. However, remote sensing images often contain noise, occluded regions, untextured areas, and repeated textures, which can lead to reduced accuracy in stereo matching and affect the quality of 3D reconstruction results. To reduce the impact of complex scenes in remote sensing images on stereo matching and to ensure both speed and accuracy, we propose a new end-to-end stereo matching network based on convolutional neural networks (CNNs). The proposed stereo matching network can learn features at different scales from the original images and construct cost volumes with varying scales to obtain richer scale information. Additionally, when constructing the cost volume, we introduce negative disparity to adapt to the common occurrence of both negative and non-negative disparities in remote sensing stereo image pairs. For cost aggregation, we employ a 3D convolution-based encoder–decoder structure that allows the network to adaptively aggregate information. Before feature aggregation, we also introduce an attention module to retain more valuable feature information, enhance feature representation, and obtain a higher-quality disparity map. By training on the publicly available US3D dataset, we obtain an accuracy of 1.115 pixels in end-point error (EPE) and 5.32% in the error pixel ratio (D1) on the test dataset, and the inference speed is 92 ms. Comparing our model with existing state-of-the-art models, we achieve higher accuracy, and the network is beneficial for the three-dimensional reconstruction of remote sensing images.

IMPROVING PAIRWISE DSM WITH 3SGM: A SEMANTIC SEGMENTATION FOR SGM USING AN AUTOMATICALLY REFINED NEURAL NETWORK

Abstract. The amount of very high resolution optical satellite images at our disposal is continuously increasing. Besides, associated satellite programs often come with high revisit rates and geometric properties that allow for either opportunistic or by-design 3D stereo reconstruction. Digital Surface Models (DSM) computed from these satellite images offer new possibilities. In the past, the high revisit rate has largely benefited glacier monitoring studies. Now, DSM with increased resolution provided on urban areas can be used for smart city applications as well. However, most of these require 3D modeling of buildings with level of details ranging from 0 to 2. This is where the need for better reconstructed buildings inside DSM arises. Indeed, building edges and corners tend to be smoothed and softened by the stereo matching step of a DSM computation pipeline. This undesired behavior can mostly be linked to the difficult task of optimizing the Disparity Space Image, thus finding good balance between smoothing untextured areas while conserving sharp discontinuities where needed. In this paper, we show how the optimization can benefit from an input building semantic segmentation. We also provide a method to create it from a very high satellite image in epipolar geometry using a convolutional neural network. To help our network generalize well on unseen areas we propose an interactive learning method based on clicked annotations. Eventually, we show that annotations can be automatically created, hence removing the need for an operator and making our solution suitable for operational conditions.

Instant Panoramic Texture Mapping with Semantic Object Matching for Large-Scale Urban Scene Reproduction.

This paper proposes a novel panoramic texture mapping-based rendering system for real-time, photorealistic reproduction of large-scale urban scenes at a street level. Various image-based rendering (IBR) methods have recently been employed to synthesize high-quality novel views, although they require an excessive number of adjacent input images or detailed geometry just to render local views. While the development of global data, such as Google Street View, has accelerated interactive IBR techniques for urban scenes, such methods have hardly been aimed at high-quality street-level rendering. To provide users with free walk-through experiences in global urban streets, our system effectively covers large-scale scenes by using sparsely sampled panoramic street-view images and simplified scene models, which are easily obtainable from open databases. Our key concept is to extract semantic information from the given street-view images and to deploy it in proper intermediate steps of the suggested pipeline, which results in enhanced rendering accuracy and performance time. Furthermore, our method supports real-time semantic 3D inpainting to handle occluded and untextured areas, which appear often when the user's viewpoint dynamically changes. Experimental results validate the effectiveness of this method in comparison with the state-of-the-art approaches. We also present real-time demos in various urban streets.

Group‐Based Atrous Convolution Stereo Matching Network

Stereo matching is the key technology in stereo vision. Given a pair of rectified images, stereo matching determines correspondences between the pair images and estimate depth by obtaining disparity between corresponding pixels. The current work has shown that depth estimation from a stereo pair of images can be formulated as a supervised learning task with an end‐to‐end frame based on convolutional neural networks (CNNs). However, 3D CNN puts a great burden on memory storage and computation, which further leads to the significantly increased computation time. To alleviate this issue, atrous convolution was proposed to reduce the number of convolutional operations via a relatively sparse receptive field. However, this sparse receptive field makes it difficult to find reliable corresponding points in fuzzy areas, e.g., occluded areas and untextured areas, owing to the loss of rich contextual information. To address this problem, we propose the Group‐based Atrous Convolution Spatial Pyramid Pooling (GASPP) to robustly segment objects at multiple scales with affordable computing resources. The main feature of the GASPP module is to set convolutional layers with continuous dilation rate in each group, so that it can reduce the impact of holes introduced by atrous convolution on network performance. Moreover, we introduce a tailored cascade cost volume in a pyramid form to reduce memory, so as to meet real‐time performance. The group‐based atrous convolution stereo matching network is evaluated on the street scene benchmark KITTI 2015 and Scene Flow and achieves state‐of‐the‐art performance.

Semi-dense and robust image registration by shift adapted weighted aggregation and variational completion

This work presents a novel approach for both stereo and optical flow that deals with large displacements, depth/motion discontinuities and occlusions. The proposed method comprises two main steps. First, a novel local stereo matching algorithm is presented, whose main novelty relies in the block-matching aggregation step. We adopt an adaptive support weights approach in which the weight distribution favors pixels that share the same displacement with the reference one. State-of-the-art methods make the weight function depend only on image features. On the contrary, the proposed weight function depends additionally on the tested shift, by giving more importance to those pixels in the block-matching with smaller cost, as these are supposed to have the tested displacement. Moreover, the method is embedded into a pyramidal procedure to locally limit the search range, which helps to reduce ambiguities in the matching process and saves computational time. Second, the non-dense local estimation is filtered and interpolated by means of a new variational formulation making use of intermediate scale estimates of the local procedure. This permits to keep the fine details estimated at full resolution while being robust to noise and untextured areas using estimates at coarser scales. The introduced variational formulation as well as the block-matching algorithm are robust to illumination changes. We test our algorithm for both stereo and optical flow public datasets showing competitive results.

Stereo Correspondence with Occlusion Handling in a Symmetric Patch-Based Graph-Cuts Model

A novel patch-based correspondence model is presented in this paper. Many segment-based correspondence approaches have been proposed in recent years. Untextured pixels and boundaries of discontinuities are imposed with hard constraints by the discontinuity assumption that large disparity variation only happens at the boundaries of segments in the above approaches. Significant improvements on performance of untextured and discontinuity area have been reported. But, the performance near occlusion is not satisfactory because a segmented region in one image may be only partially visible in the other one. To solve this problem, we utilize the observation that the shared edge of a visible area and an occluded area corresponds to the discontinuity in the other image. So, the proposed model conducts color segmentation on both images first and then a segment in one image is further cut into smaller patches corresponding to the boundaries of segments in the other when it is assigned with a disparity. Different visibility of patches in one segment is allowed. The uniqueness constraint in a segment level is used to compute the occlusions. An energy minimization framework using graph-cuts is proposed to find a global optimal configuration including both disparities and occlusions. Besides, some measurements are taken to make our segment-based algorithm suffer less from violation of the discontinuity assumption. Experimental results have shown superior performance of the proposed approach, especially on occlusions, untextured areas, and near discontinuities.