Dense Matching Research Articles

Dual Guided Aggregation Network for Stereo Image Matching.

Stereo image dense matching, which plays a key role in 3D reconstruction, remains a challenging task in photogrammetry and computer vision. In addition to block-based matching, recent studies based on artificial neural networks have achieved great progress in stereo matching by using deep convolutional networks. This study proposes a novel network called a dual guided aggregation network (Dual-GANet), which utilizes both left-to-right and right-to-left image matchings in network design and training to reduce the possibility of pixel mismatch. Flipped training with a cost volume consistentization is introduced to realize the learning of invisible-to-visible pixel matching and left–right consistency matching. In addition, suppressed multi-regression is proposed, which suppresses unrelated information before regression and selects multiple peaks from a disparity probability distribution. The proposed dual network with the left–right consistent matching scheme can be applied to most stereo matching models. To estimate the performance, GANet, which is designed based on semi-global matching, was selected as the backbone with extensions and modifications on guided aggregation, disparity regression, and loss function. Experimental results on the SceneFlow and KITTI2015 datasets demonstrate the superiority of the Dual-GANet compared to related models in terms of average end-point-error (EPE) and pixel error rate (ER). The Dual-GANet with an average EPE performance = 0.418 and ER (>1 pixel) = 5.81% for SceneFlow and average EPE = 0.589 and ER (>3 pixels) = 1.76% for KITTI2005 is better than the backbone model with the average EPE performance of = 0.440 and ER (>1 pixel) = 6.56% for SceneFlow and average EPE = 0.790 and ER (>3 pixels) = 2.32% for KITTI2005.

Influence of Block Geometry Configuration on Multi-Image Dense Matching

Digital photogrammetry is a widespread surveying technique in different fields of application due to its flexibility, versatility and cost-effectiveness. Despite its increasing automation and simplicity, a proper image block design is crucial to ensure high standards of performance and accuracy. Studies on camera network design have been largely dealt with in the scientific literature with reference to image orientation process, while they are still poor on dense matching. This paper investigates the influence of different block geometry configurations on multi-image dense matching. Starting from the same orientation solution, dense matching was performed considering different combinations of number of images and base length distance between the first and the last image within a strip. The raster Digital Elevation Models (DEM) resulting from each sequence of images were compared with a reference DEM to assess accuracy and completeness. The tests were conducted using different cameras and at various test sites to assess different survey conditions and generalize the findings. The presented results provide some operational guidance on block geometry optimization to maximize the accuracy and completeness.

Registration of Terrestrial LiDAR and Panoramic Imagery Using the Spherical Epipolar Line and Spherical Absolute Orientation Model

Registration between terrestrial LiDAR and optical imagery plays a crucial role in information fusion. However, it is difficult to find reliable correlations among the different feature information of optical imagery and LiDAR point clouds. Therefore, in order to achieve high-precision registration of heterogeneous sensors, a method based on spherical epipolar line and spherical absolute orientation is proposed in this paper. The method firstly projects the LiDAR point clouds into spherical images based on the spherical imaging model and derives the spherical epipolar line equation. Then the relative and absolute orientations of the spherical LiDAR images and the optical images are performed based on manually selected control points. Finally, based on Harris corner extraction, combined with the geometric constraints of the spherical epipolar line and absolute orientation, dense matching between optical and LIDAR images are achieved, and all matching points are used as control points for registration to improve the accuracy of manually selected points registration. Multiple sets of test data are acquired outdoors using a FARO Focus S laser scanner, a Z + F IMAGER 5010C laser scanner, and a Ladybug5+ panoramic camera. The experimental results show that the method in this paper is practical and improves the accuracy of manual points selection registration, and the degree of improvement is related to the number of successfully matched corner points.

Decennial Geomorphic Transport From Archived Time Series Digital Elevation Models: A cookbook for tropical and alpine environments

On the seasonal timescale, for accessible locations and when manpower is available, direct observations and field surveys are the most useful and standard approaches. However very limited studies have been conducted through direct examination on decennial-to-century timescales due to observational constrains. Here, we present an open and reproducible pipeline based on historical aerial images (across a timeline of up to 70 years) that includes sensor calibration, dense matching, and elevation reconstruction in two areas of interest that represent pristine examples of tropical and alpine environments. The Remparts and Langevin rivers, on Réunion Island, and the Bossons glacier, in the French Alps, share limited accessibility (in time and space) that can be overcome only by remote sensing. We reach a metric-to-submetric resolution close to the nominal image spatial sampling. This provides elevation time series with better resolution than most recent satellite images, such as Pleiades, in a decennial time period.

Binocular Images Dense Matching considering Image Adaptive Color Weights and Feature Points

When the matching cost function in Semiglobal Matching is unstable, the inaccurate matching cost values will be propagated in the cost aggregation process. It will lead to a serious mismatching phenomenon. To address the problem, a binocular images dense matching method considering image adaptive color weights and feature points was proposed. Firstly, The Color Birchfield Tomasi (CBT) matching cost calculation method was proposed to obtain a stable initial cost volume, which combined image adaptive color weights and gradient information. Secondly, the Scale-invariant Feature Transform matching algorithm was used to extract the a priori feature points from binocular images. Then, the feature points were filtrated. The cost volume was optimized by using their coordinate information and disparity information. Finally, an aggregation path segmentation rectification method was adopted to optimize the SGM aggregation paths and reduce the propagation of incorrect paths. Experimental results demonstrate that the proposed method can effectively improve the stability and accuracy of dense matching, reduce the mismatching phenomenon, and finally produce high-quality disparity maps.

EXPLORING THE POTENTIALS OF UAV PHOTOGRAMMETRIC POINT CLOUDS IN FAÇADE DETECTION AND 3D RECONSTRUCTION OF BUILDINGS

Abstract. The use of Airborne Laser Scanner (ALS) point clouds has dominated 3D buildings reconstruction research, thus giving photogrammetric point clouds less attention. Point cloud density, occlusion and vegetation cover are some of the concerns that promote the necessity to understand and question the completeness and correctness of UAV photogrammetric point clouds for 3D buildings reconstruction. This research explores the potentials of modelling 3D buildings from nadir and oblique UAV image data vis a vis airborne laser data. Optimal parameter settings for dense matching and reconstruction are analysed for both UAV image-based and lidar point clouds. This research employs an automatic data driven model approach to 3D building reconstruction. A proper segmentation into planar roof faces is crucial, followed by façade detection to capture the real extent of the buildings’ roof overhang. An analysis of the quality of point density and point noise, in relation to setting parameter indicates that with a minimum of 50 points/m2, most of the planar surfaces are reconstructed comfortably. But for smaller features than dormers on the roof, a denser point cloud than 80 points/m2 is needed. 3D buildings from UAVs point cloud can be improved by enhancing roof boundary by use of edge information from images. It can also be improved by merging the imagery building outlines, point clouds roof boundary and the walls outline to extract the real extent of the building.

A NOVEL INTRINSIC IMAGE DECOMPOSITION METHOD TO RECOVER ALBEDO FOR AERIAL IMAGES IN PHOTOGRAMMETRY PROCESSING

Abstract. Recovering surface albedos from photogrammetric images for realistic rendering and synthetic environments can greatly facilitate its downstream applications in VR/AR/MR and digital twins. The textured 3D models from standard photogrammetric pipelines are suboptimal to these applications because these textures are directly derived from images, which intrinsically embedded the spatially and temporally variant environmental lighting information, such as the sun illumination, direction, causing different looks of the surface, making such models less realistic when used in 3D rendering under synthetic lightings. On the other hand, since albedo images are less variable by environmental lighting, it can, in turn, benefit basic photogrammetric processing. In this paper, we attack the problem of albedo recovery for aerial images for the photogrammetric process and demonstrate the benefit of albedo recovery for photogrammetry data processing through enhanced feature matching and dense matching. To this end, we proposed an image formation model with respect to outdoor aerial imagery under natural illumination conditions; we then, derived the inverse model to estimate the albedo by utilizing the typical photogrammetric products as an initial approximation of the geometry. The estimated albedo images are tested in intrinsic image decomposition, relighting, feature matching, and dense matching/point cloud generation results. Both synthetic and real-world experiments have demonstrated that our method outperforms existing methods and can enhance photogrammetric processing.

INVESTIGATING SPHERICAL EPIPOLAR RECTIFICATION FOR MULTI-VIEW STEREO 3D RECONSTRUCTION

Abstract. Multi-view stereo (MVS) reconstruction is essential for creating 3D models. The approach involves applying epipolar rectification followed by dense matching for disparity estimation. However, existing approaches face challenges in applying dense matching for images with different viewpoints primarily due to large differences in object scale. In this paper, we propose a spherical model for epipolar rectification to minimize distortions caused by differences in principal rays. We evaluate the proposed approach using two aerial-based datasets consisting of multi-camera head systems. We show through qualitative and quantitative evaluation that the proposed approach performs better than frame-based epipolar correction by enhancing the completeness of point clouds by up to 4.05% while improving the accuracy by up to 10.23% using LiDAR data as ground truth.

GA-Net-Pyramid: An Efficient End-to-End Network for Dense Matching

Dense matching plays a crucial role in computer vision and remote sensing, to rapidly provide stereo products using inexpensive hardware. Along with the development of deep learning, the Guided Aggregation Network (GA-Net) achieves state-of-the-art performance via the proposed Semi-Global Guided Aggregation layers and reduces the use of costly 3D convolutional layers. To solve the problem of GA-Net requiring large GPU memory consumption, we design a pyramid architecture to modify the model. Starting from a downsampled stereo input, the disparity is estimated and continuously refined through the pyramid levels. Thus, the disparity search is only applied for a small size of stereo pair and then confined within a short residual range for minor correction, leading to highly reduced memory usage and runtime. Tests on close-range, aerial, and satellite data demonstrate that the proposed algorithm achieves significantly higher efficiency (around eight times faster consuming only 20–40% GPU memory) and comparable results with GA-Net on remote sensing data. Thanks to this coarse-to-fine estimation, we successfully process remote sensing datasets with very large disparity ranges, which could not be processed with GA-Net due to GPU memory limitations.

Densely Enhanced Semantic Network for Conversation System in Social Media

The human–computer conversation system is a significant application in the field of multimedia. To select an appropriate response, retrieval-based systems model the matching between the dialogue history and response candidates. However, most of the existing methods cannot fully capture and utilize varied matching patterns, which may degrade the performance of the systems. To address the issue, a densely enhanced semantic network (DESN) is proposed in our work. Given a multi-turn dialogue history and a response candidate, DESN first constructs the semantic representations of sentences from the word perspective, the sentence perspective, and the dialogue perspective. In particular, the dialogue perspective is a novel one introduced in our work. The dependencies between a single sentence and the whole dialogue are modeled from the dialogue perspective. Then, the response candidate and each utterance in the dialogue history are made to interact with each other. The varied matching patterns are captured for each utterance–response pair by using a dense matching module. The matching patterns of all the utterance–response pairs are accumulated in chronological order to calculate the matching degree between the dialogue history and the response. The responses in the candidate pool are ranked with the matching degree, thereby returning the most appropriate candidate. Our model is evaluated on the benchmark datasets. The experimental results prove that our model achieves significant and consistent improvement when compared with other baselines.

A Transformer-Based Coarse-to-Fine Wide-Swath SAR Image Registration Method under Weak Texture Conditions

As an all-weather and all-day remote sensing image data source, SAR (Synthetic Aperture Radar) images have been widely applied, and their registration accuracy has a direct impact on the downstream task effectiveness. The existing registration algorithms mainly focus on small sub-images, and there is a lack of available accurate matching methods for large-size images. This paper proposes a high-precision, rapid, large-size SAR image dense-matching method. The method mainly includes four steps: down-sampling image pre-registration, sub-image acquisition, dense matching, and the transformation solution. First, the ORB (Oriented FAST and Rotated BRIEF) operator and the GMS (Grid-based Motion Statistics) method are combined to perform rough matching in the semantically rich down-sampled image. In addition, according to the feature point pairs, a group of clustering centers and corresponding images are obtained. Subsequently, a deep learning method based on Transformers is used to register images under weak texture conditions. Finally, the global transformation relationship can be obtained through RANSAC (Random Sample Consensus). Compared with the SOTA algorithm, our method’s correct matching point numbers are increased by more than 2.47 times, and the root mean squared error (RMSE) is reduced by more than 4.16%. The experimental results demonstrate that our proposed method is efficient and accurate, which provides a new idea for SAR image registration.

Jitter Detection Method Based on Sequence CMOS Images Captured by Rolling Shutter Mode for High-Resolution Remote Sensing Satellite

Satellite platform jitter is a non-negligible factor that affects the image quality of optical cameras. Considering the limitations of traditional platform jitter detection methods that are based on attitude sensors and remote sensing images, this paper proposed a jitter detection method using sequence CMOS images captured by rolling shutter for high-resolution remote sensing satellite. Through the three main steps of dense matching, relative jitter error analysis, and absolute jitter error modeling using sequence CMOS images, the periodic jitter error on the imaging focal plane of the spaceborne camera was able to be measured accurately. The experiments using three datasets with different jitter frequencies simulated from real remote sensing data were conducted. The experimental results showed that the jitter detection method using sequence CMOS images proposed in this paper can accurately recover the frequency, amplitude, and initial phase information of satellite jitter at 100 Hz, 10 Hz, and 2 Hz. Additionally, the detection accuracy reached 0.02 pixels, which can provide a reliable data basis for remote sensing image jitter error compensation.

Quasi-Dense Matching Algorithm for Close-Range Image Combined With Feature Line Constraint

Point-based sparse or dense matching can typically obtain satisfactory 3D point clouds of general contour features, but the deformation problem at the edges of artificial objects is prominent. Thus, a quasi-dense matching algorithm for close-range images combined with feature line constraint is proposed in this study. The method utilizes reliable matched points to construct the initial Delaunay triangulation and then optimizes the triangulation using the matched feature line. On this basis, iterative quasi-dense matching based on triangulation constraint is implemented. The process achieves matching with the center of the inscribed circle of each triangle as the seed point and growing matching. Two sets of stereo image pairs acquired using smartphones and four sets of sequence images provided by public datasets are selected for quasi-dense matching experiments. The comparison of results of constraint matching of the two triangulations before and after optimization as well as the matching results obtained via VisualSFM software demonstrated that the 3D point cloud obtained via quasi-dense matching with feature line constraint presents better results at the edges of buildings, thereby confirming the effectiveness of the proposed algorithm.

Research on 3D Modeling of Wupaolong based on Sparse Point Cloud Reconstruction of SFM Algorithm

As a classic project for the inheritance of Chinese folk culture, Wupaolong has the characteristics of unique material selection and complex production technology. However, as the finished products of folk handicrafts may not comply with production standards, a common problem in the production of handicrafts. Focusing on the problems of poor visual effects such as uneven surface and jagged edges in the craft paper binding model constructed by point cloud, and the need to enhance the sense of reality, this paper proposes an optimization method for 3D model and triangulation based on 3D line features. This method first preprocesses the feature points and performs sparse matching and dense matching on the processed feature points to generate a dense point cloud. The dense point cloud is then processed by an adaptive weighted median filtering method. This is followed by application of surface modeling method based on Delaunay algorithm to realize the reconstruction of object triangulation. Based on the triangular patch, the reference plane of the craft paper binding surface is fitted, and the model surface is corrected and optimized based on the reference plane. With the aid of three-dimensional line segments, the triangulation at the edge of the craft paper binding is corrected and optimized. Finally, an optimized three-dimensional model of craft paper binding is obtained. Experimental results show that the proposed algorithm can improve the unevenness of the 3D model plane and edges, and maintain the characteristics of the 3D model plane and edges. Compared with the existing surface reconstruction algorithms based on point cloud features, the optimized triangulation and model have higher accuracy and quality, which can greatly improve the visual effect of the model and improve the accuracy of the three-dimensional model. The research results of this paper can realize 3D modeling development for the production process of Wupaolong, effectively improve the elevation accuracy of the model, realize the production standard of digital Wupaolong model, and provide reference for folk production process.

Global Color Consistency Correction for Large-Scale Images in 3-D Reconstruction

Global color consistency correction for multiview images in three-dimensional (3-D) reconstruction is an important problem. The color differences between the images will affect the result of dense matching, thereby reducing the geometric accuracy of the mesh model. Moreover, it will also affect the result of texture mapping, causing color differences in the textured model. The color correction method based on global optimization is mainly used to solve this problem. And existing methods usually use sparse matching points as the color correspondences, but the correction results are not accurate enough as a result of the sparsity of the points. Besides, their efficiency of solving large-scale images globally is low. This article proposes a novel color correction method to eliminate the color differences between large-scale multiview images effectively. The core idea of our method is to group images by graph partition algorithm, and then perform intragroup correction and intergroup correction in sequence. First, for each pair of images, we calculate the reliable matching regions around the sparse points as the color correspondences according to the local homography principle. Compared with sparse matching points, our strategy can achieve more accurate color correction results. Next, for large-scale images, we partition them into many groups. For each group of images, the correction parameters are solved to eliminate the color differences of the images included in the group. Finally, we eliminate the color differences between groups by intergroup correction to achieve overall color consistency. Experimental results on typical datasets demonstrate that the proposed method is better than the current representative methods. The proposed method shows better color consistency in the extreme cases, and also exhibits higher computational efficiency on large-scale image sets.

A Stereo Matching Method for 3D Image Measurement of Long-Distance Sea Surface

Tsunamis are some of the most destructive natural disasters. Some proposed tsunami measurement and arrival prediction systems use a limited number of instruments, then judge the occurrence of the tsunami, forecast its arrival time, location and scale. Since there are a limited number of measurement instruments, there is a possibility that large prediction errors will occur. In order to solve this problem, a long-distance tsunami measurement system based on the binocular stereo vision principle is proposed in this paper. The measuring range is 4–20 km away from the system deployment site. In this paper, we will focus on describing the stereo matching method for the proposed system. This paper proposes a two-step matching method. It first performs fast sparse matching, and then complete high precision dense matching based on the results of the sparse matching. A matching descriptor based on the physical features of sea waves is proposed to solve the matching difficulty caused by the similarity of sea surface image textures. The relationship between disparity and the y coordinate is built to reduce the matching search range. Experiments were conducted on sea surface images with different shooting times and distances; the results verify the effectiveness of the presented method.

A General Framework of Remote Sensing Epipolar Image Generation

Epipolar images can improve the efficiency and accuracy of dense matching by restricting the search range of correspondences from 2-D to 1-D, which play an important role in 3-D reconstruction. As most of the satellite images in archives are incidental collections, which do not have rigorous stereo properties, in this paper, we propose a general framework to generate epipolar images for both in-track and cross-track stereo images. We first investigate the theoretical epipolar constraints of single-sensor and multi-sensor images and then introduce the proposed framework in detail. Considering large elevation changes in mountain areas, the publicly available digital elevation model (DEM) is applied to reduce the initial offsets of two stereo images. The left image is projected into the image coordinate system of the right image using the rational polynomial coefficients (RPCs). By dividing the raw images into several blocks, the epipolar images of each block are parallel generated through a robust feature matching method and fundamental matrix estimation, in which way, the horizontal disparity can be drastically reduced while maintaining negligible vertical disparity for epipolar blocks. Then, stereo matching using the epipolar blocks can be easily implemented and the forward intersection method is used to generate the digital surface model (DSM). Experimental results on several in-track and cross-track images, including optical-optical, SAR-SAR, and SAR-optical pairs, demonstrate the effectiveness of the proposed framework, which not only has obvious advantages in mountain areas with large elevation changes but also can generate high-quality epipolar images for flat areas. The generated epipolar images of a ZiYuan-3 pair in Songshan are further utilized to produce a high-precision DSM.

Part-to-full shape matching of different human subjects

Shape matching is a fundamental operation in digital geometry processing and computer graphics. Challenges in shape matching include finding correspondences of partial shapes with deformations, as well as topological noise and ambiguities. This paper presents a partial shape correspondence algorithm based on the concept of the functional map. An iterative dense matching algorithm, incorporating sparse and guided dense matching, is proposed along with a new objective function including both descriptor matching error and transformation error. Rank estimation with the rank direction is proposed to achieve more accurate slope approximation of the functional map. The slope is beneficial because it directly influences the matching efficiency. The experimental results obtained using FAUST and SHREC′16 datasets demonstrate the effectiveness of our proposed algorithm for matching the shapes of different human subjects and shapes with large missing parts compared with state-of-the art algorithms. The proposed algorithm provides an average geodesic distance of <0.033 even when the missing part is up to 80% of the area.

DOCUMENTATION AND HBIM OF INDUSTRIAL HERITAGE USING DRONE IMAGES: PETROLEUM RESERVOIR OF MASHHAD

Abstract. Documentation is a key step for cognition of cultural heritage, and also a requisite for conservation and adaptive reuse actions. Petroleum reservoir of Mashhad (constructed in 1925) was documented by means of Drone images. The photogrammetric documentation was aimed at provision of 3D models and as a dataset for the creation of BIM models. These data-enriched models could be used on Digital Twin platforms for monitoring and operational purposes, a concept that is becoming increasingly relevant in the field of cultural heritage conservation. The discussion of this process demonstrates that the dense matching of drone images may generate centimeter-level precision and can provide a proper basis for BIM and Digital Twin platforms. The capability of this system will help the community in sustainable development in order to preserve the monuments and determine the appropriate urban use in heritage buildings.

VIRTUAL RESTORATION OF MISSING PAINT LOSS OF MURAL BASED ON GENERATIVE ADVERSARIAL NETWORK

Abstract. Mural painting is one of the important cultural heritage reflecting the historical migration of the nation. In order to inherit these precious historical and cultural heritage, how to non - destructively and digitally protect and restore the existing murals has become an urgent task. The use of computer - assisted restoration of murals can not only save manpower and material resources, but also avoid secondary damage to the murals.However, most of the existing computer-assisted mural restoration algorithms use similar blocks with priority calculations and matching blocks in adjacent areas to guide mural restoration. There are some problems such as incoherent overall semantic structure, unnatural detail texture and inability to effectively repair large area missing remain to be solved. Aiming at the problems existing in the restoration of large area diseases such as paint loss and color fading in murals, we constructed a fine image restoration network model which based on generative adversarial network. A multi-scale dense matching repair network based on a generative adversarial network is constructed. First, the dense combination of dilated convolutions is used to improve the repair effect of detailed textures, Then, mean absolute Error, (Visual Geometry Group, VGG) feature matching, auto-guided regression, and geometric alignment are used as the loss function to guide the training of the generative network. Second, the discriminator with local and global branches is used to train the discriminant network, so that the repaired image is in the local and global content. Experiments were performed on the three mural data sets one by one. The results show that the network model can effectively restore the lines and faces in the murals. The images restored are not only coherent in semantic details, but also natural in color, which is conducive to the appreciation and display of murals. Thus, as one of the important directions of cultural heritage digital protection，the use of generative adversarial network in the digital restoration of ancient murals have been proved to be effective. It not only provides a reference for the true restoration of the murals but also means a lot to the preservation of murals.