Area-based Matching Research Articles

SAR-optical feature matching: A large-scale patch dataset and a deep local descriptor

Image matching is a key prerequisite for image fusion. Currently, deep learning methods had shown great potential in matching. However, these methods mainly focus on optical images or area-based matching and their performance highly relies on the diversity of the training sets. This paper makes two major contributions to the synthetic aperture radar (SAR)-optical image feature matching task. First, we create a high-quality open SAR and optical patch dataset called SOPatch, which contains more than 650000 matching patch pairs. SOPatch is generated from a variety of satellite images (Sentinel-1, Sentinel-2, Gaofen-3, etc.) and contains a rich set of features (mountains, lakes, buildings, farmland, bare land, etc.), which can provide robustness to sensor and localization changes and train a model with good generalization ability. Similar to the HPatches (Balntas et al., 2017), we also add position and rotation noise to make the dataset more realistic and robust. Second, a local descriptor for SAR-optical matching called SODescNet is proposed. We first weaken the speckle noise of SAR by multiscale dilated convolution and channel attention mechanisms and then use DenseNet-CSP (cross stage partial) as the backbone for feature description. To reduce the difficulty of network training, we first use a Siamese network for pre-training and then fine-tune it via a pseudo-Siamese network. Extensive experiments show that our SODescNet is comparable or superior to state-of-the-arts.

A Discriminative Image Registration Model with Hybrid Feature Detector

Image registration is a manner of aligning two or more images of the same scene which achieves maximum accuracy. Analysis of geometric distortion, caused by the difference in viewing angle, sensor resolution and distance. The critical step in image registration is the collection of feature points and evaluate a spatial transformation mainly when outliers are present. In our proposed framework we have used the hybrid feature-based and area-based methods together. Before applying those methods, the Bilateral Filter is applied for the preprocessing of both the images, the reference image and the image that needs to be registered in order to make the images noise free. For feature detection, the ORB (Oriented FAST and Rotated BRIEF) is used which is basically a fusion of FAST key point detector and BRIEF descriptor with many modifications to enhance the performance. KNN (K-Nearest Neighbors) is used for matching similar points and for reducing the miss matches the efficient algorithm is used which is Random Sample Consensus (RANSAC). The template matching method is applied for the area-based matching.

Multi-temporal orthophoto and digital surface model registration produced from UAV imagery over an agricultural field

Correcting the three-dimensional geometric error is essential to effectively use the multi-temporal unmanned aerial vehicle (UAV) orthophoto and digital surface model (DSM) acquired from the agricultural field. Although ground control points (GCPs) obtained through field surveys are usually used to calibrate geometrical errors establishing/maintaining GCPs and surveying them in the field are time-consuming and inefficient. Therefore, we propose a simple and efficient methodology to improve the geometric registration of multi-temporal orthophotos and DSMs without GCPs. In the proposed method, coarse to fine image registration is performed first, which corrects severe to slight errors by sequential feature and area-based matching methods. Subsequently, we extract height-invariant regions in multi-temporal DSM pairs, called elevation invariant feature (EIF), using the EIFs to register DSMs by estimating a linear regression model. Various experiments were conducted to analyze the absolute and relative accuracies using ten multi-temporal orthophotos and DSMs, and the robustness of the proposed method was evaluated using data obtained from another site. The experimental results demonstrate that the geometric quality of registered orthophotos and DSMs was significantly improved.

Mechanical System and Template-Matching-Based Position-Measuring Method for Automatic Spool Positioning and Loading in Welding Wire Winding

Welding wire is a major type of welding consumable, which needs to be winded onto spools for sale. Currently, the winding process is accomplished manually due to obstacles such as automatic spool loading and clamping. When loading the spool, the angular position of the spool is a prerequisite for matching the drive rod on the spindle and drive bore on the spool. Therefore, this paper proposes a template-matching method combined with area-based matching and feature-point detection to measure the angular position of the spool, and presents a mechanical system that can rotate the spool to match the drive rod and push the spool onto the spindle. A novel feature-point distribution density (FPDD) method was developed to accelerate the matching process and improve matching reliability by pre-locating the searching area. The robustness and accuracy of the template-matching-based measuring method were validated using a built prototype of the mechanical system. The comparison result shows that the proposed method was superior in robustness, accuracy, and speed, and it was efficient for automatic spool loading in the welding wire winding process.

Using Total Probability in Image Template Matching

Image template matching is a main task in photogrammetry and computer vision. The matching can be used to automatically determine the 3D coordinates of a point. A firstborn image matching method in fields of photogrammetry and computer vision is area-based matching, which is based on correlation measuring that uses normalised cross-correlation. However, this method fails at a discontinuous edge and at the area of low illumination or at geometric distortion because of changes in imaging location. Thus, these points are considered outliers. The proposed method measures correlations, which is based on normalised cross-correlation, at each point by using various sizes of window and then considering the probability of correlations for each window. Thereafter, the determined probability values are integrated. On the basis of a specific threshold value, the point of maximum total probability correlation is recognised as a corresponding point. The algorithm is applied to aerial images for Digital Surface Model (DSM) generation. Results show that the corresponding points are identified successfully at different locations, especially at a discontinuous point, and that a Digital Surface Model of high resolution is generated.

TIE POINT GENERATION IN HYPERSPECTRAL CUBES FOR ORIENTATION WITH POLYNOMIAL MODELS

Abstract. This paper presents a technique for tie point generation in hyperspectral images collected by a camera with time-sequential principle for band acquisition (i.e., non-synchronized bands). In mobile applications, each band is acquired at a different time, which generates different camera positions and attitude angles. Due to the large number of bands, a bundle adjustment with polynomial models can be applied to sample bands and then, EOP of other bands are interpolated. The determination of homologue points in all sample bands is required to ensure geometric robustness. A procedure was developed to extract tie points from a reference band which were then transferred to the other sample bands. The technique uses a Helmert geometric transformation combined with majority voting to estimate point transfer functions, followed by area-based matching. Experiments with image orientation were conducted to apply and assess the technique. The tests showed an increase in height discrepancy when tie points are not located in all sample bands highlighting the relevance of the proposed filtering. The accuracy of the technique achieved less than 1 GSD in planimetry and 2 GSD in altimetry using the tie points with the maximum number of rays. Thus, the polynomial approach enables interpolation of other bands according to the parameters of the polynomial function.

High-Resolution Optical Remote Sensing Image Registration via Reweighted Random Walk Based Hyper-Graph Matching

High-resolution optical remote sensing image registration is still a challenging task due to non-linearity in the intensity differences and geometric distortion. In this paper, an efficient method utilizing a hyper-graph matching algorithm is proposed, which can simultaneously use the high-order structure information and radiometric information, to obtain thousands of feature point pairs for accurate image registration. The method mainly consists of the following steps: firstly, initial matching by Uniform Robust Scale-Invariant Feature Transform (UR-SIFT) is carried out in the highest pyramid image level to derive the approximate geometric relationship between the images; secondly, two-stage point matching is performed to find the matches, that is, a rotation and scale invariant area-based matching method is used to derive matching candidates for each feature point and an efficient hyper-graph matching algorithm is applied to find the best match for each feature point; thirdly, a local quadratic polynomial constraint framework is used to eliminate match outliers; finally, the above process is iterated until finishing the matching in the original image. Then, the obtained correspondences are used to perform the image registration. The effectiveness of the proposed method is tested with six pairs of high-resolution optical images, covering different landscape types—such as mountain area, urban, suburb, and flat land—and registration accuracy of sub-pixel level is obtained. The experiments show that the proposed method outperforms the conventional matching algorithms such as SURF, AKAZE, ORB, BRISK, and FAST in terms of total number of correct matches and matching precision.

COMPARING STEREO IMAGE MATCHING PERFORMANCE BY MULTIDIMENSIONAL SEARCH WINDOWS

Abstract. Image matching is a key technology for extraction of dense point cloud and 3D terrain information using satellite/aerial imagery. In image matching using brightness values of pixels, the size of search window is an important factor for determining the matching performance. In this study, we perform matching using multi-dimensional search windows applicable to area-based matching and compare the performance. Also, the search window is reconfigured by using the linear information existing on the image, and the matching is tried. Comparing the fixed search window and the multi-window matching results, it was confirmed that the multiple windows under the same conditions show relatively high accuracy. We can also see that the method of applying the line element has slightly better accuracy. As a result of applying the line element extraction technique, a large number of pixels are not extracted compared with the total image pixel amount. There was no significant difference in the results of visual analysis. However, we have confirmed that this technique has contributed to improving accuracy.

FAST AND ROBUST STEREO MATCHING ALGORITHM FOR OBSTACLE DETECTION IN ROBOTIC VISION SYSTEMS

In this paper, we propose a new area-based stereo matching method by improving the classical Census transform. It is a difficult task to match the corresponding points in two images taken by stereo cameras, mostly under variant illumination and non-ideal conditions. The classic Census nonparametric transform offers some improvements in the accuracy of disparity map in these conditions but it also has some disadvantages. Because of the complexity of the algorithm, the performance is not suitable for real-time robotic systems. In order to solve this problem, this paper presents the differential transform using Maximum intensity differences of the pixel placed in the center of a defined window and the pixel in the neighborhood to reduce complexity and obtain better performance compared to the Census transform. Experimental results show that the proposed method, achieves better efficiency in terms of speed and memory consumption. Moreover, we have added a new feature to widen the depth detection range. With the help of the proposed method, robots can detect obstacles between 25cm to 400cm from robot cameras. The result shows that the method has the ability to work in a wide variety of lighting conditions, while the stereo matching performs the depth detection computation with speed of 30FPS.

Quality assessment of high density digital surface model over different land cover classes

Background and Purpose: Recent research on generation of digital surface models (DSMs) using image matching methods revealed a great potential of DSM application in forestry, especially in forest inventory. However, research dealing with DSM generation from digital aerial images are still lacking in Croatia. Therefore, the main objective of this study was to present the workflow for generating high density DSM from colour infrared (CIR) digital stereo aerial images using area-based image matching algorithm. Materials and Methods: The high density DSM was generated from colour infrared digital aerial stereo images using Dense DTM algorithm of PHOTOMOD software - an area-based image matching algorithm which operates on the principle of cross-correlation approach. To evaluate the quality of the generated DSM, an agreement assessment with manual stereo measurements was conducted over three different land cover classes (forests, shrubs, grasslands) using the same images as for DSM generation. Results: The good vertical agreement between the generated DSM and stereo measurement was achieved for all three land cover classes present at the research area. The highest vertical agreement was obtained for the grassland land cover class (RMSE=0.36), slightly lower for forest (RMSE=0.62), whereas the lowest vertical agreement was obtained for shrub land cover class (RMSE=0.83). Conclusions: The results of this research are very promising and suggest that the high density DSM generated from digital aerial stereo images and by using the proposed methodology has the potential to be used in forestry, primarily in forest inventory. Therefore, further research should be focused on generation of CHM by subtracting available DTM from the high density DSM and on the examination of its potential for deriving various forest attributes.

ADAPTIVE HIERARCHICAL DENSE MATCHING OF MULTI-VIEW AIRBORNE OBLIQUE IMAGERY

Abstract. Traditional single-lens vertical photogrammetry can obtain object images from the air with rare lateral information of tall buildings. Multi-view airborne photogrammetry can get rich lateral texture of buildings, while the common area-based matching for oblique images may lose efficacy because of serious geometric distortion. A hierarchical dense matching algorithm is put forward here to match two oblique airborne images of different perspectives. Based on image hierarchical strategy and matching constraints, this algorithm delivers matching results from the upper layer of the pyramid to the below and implements per-pixel dense matching in the local Delaunay triangles between the original images. Experimental results show that the algorithm can effectively overcome the geometric distortion between different perspectives and achieve pixel-level dense matching entirely based on the image space.

Strip Adjustment of Airborne Laser Scanner Data Using Area-based Surface Matching

Multiple strips are required for large area mapping using ALS (Airborne Laser Scanner) system. LiDAR (Light Detection And Ranging) data collected from the ALS system has discrepancies between strips due to systematic errors of on-board laser scanner and GPS/INS, inaccurate processing of the system calibration as well as boresight misalignments. Such discrepancies deteriorate the overall geometric quality of the end products such as DEM (Digital Elevation Model), building models, and digital maps. Therefore, strip adjustment for minimizing discrepancies between overlapping strips is one of the most essential tasks to create seamless point cloud data. This study implemented area-based matching (ABM) to determine conjugate features for computing 3D transformation parameters. ABM is a well-known method and easily implemented for this purpose. It is obvious that the exact same LiDAR points do not exist in the overlapping strips. Therefore, the term “conjugate point” means that the location of occurring maximum similarity within the overlapping strips. Coordinates of the conjugate locations were determined with sub-pixel accuracy. The major drawbacks of the ABM are sensitive to scale change and rotation. However, there is almost no scale change and the rotation angles are quite small between adjacent strips to apply AMB. Experimental results from this study using both simulated and real datasets demonstrate validity of the proposed scheme.

Measurement Error Correction of <i>y</i> Coordinate in Three Dimensional Localization of Tomatoes Using Binocular Stereo Vision

Accuracy three dimensional coordinates of fruits and vegetables are very important to harvesting robots to harvest fruits and vegetables correctly. To decrease the measurement errors of the y coordinates of tomatoes, we analyzed the measurement errors of y coordinate acquired using binocular stereo vision based on three stereo matching methods. These three stereo matching methods were centroid-based, area-based, and combination stereo matching methods. After stereo matching, the three dimensional coordinates of tomatoes could be acquired based on the triangle ranging principle. Tests of 225 pairs of stereo images of three plastic balls used as normal balls acquired at the distances from 300 to 1000 mm showed that the ranges of the measurement errors of y coordinate acquired based on three stereo matching methods changed with the image acquisition distances obviously. Moreover, the measurement errors of y coordinate appeared linear decreasing trends approximately. Therefore, binary linear regression models were set up to reduce the ranges of the measurement errors of y coordinate of three balls. These models were used as correction models of the measurement values of y coordinate and were helpful to reduce the measurement errors of y coordinate. However, there were owe correction and overcorrection conditions when the image acquisition distances were smaller and larger than 750 mm separately. Then, the correction models based on piecewise binary linear regression were used to solve this problem. The ranges of the measurement errors of y coordinate were reduced further. Tests of 225 pairs of stereo images of three tomatoes acquired at the distances from 300 to 1000 mm showed that the ranges of the measurement errors of y coordinate acquired based on three stereo matching methods were separately from [-20.9, -6.6], [-19.9, -3.44], [-19.9, -3.48] mm to [-6.84, -0.06], [-5.84, -0.82], [-5.85, -0.83] mm after the correction using the piecewise binary linear regression models. It proved that the piecewise binary linear regression models were helpful to reduce the measurement errors of y coordinate in three dimensional localization of tomatoes using binocular stereo vision.

Three-dimensional lunar surface reconstruction based on remote sensing imagery from Chang E-2

In this paper, we propose a complete solution for effective, automatic and accurate reconstruction of the lunar surface based on the linear array push-broom imagery from Chang E-2 (CE-2). First, with the sparse ephemeris data, an approach for estimating the corresponding areas between forward (F) imagery and backward (B) imagery is proposed through exploiting the imaging characteristics of linear push-broom cameras. Second, feature based matching (FBM) is conducted between F and B imagery, followed by Area Based Matching (ABM) for dense correspondence. Third, the extrinsic parameters of every scan line are estimated according to the supplied sparse ephemeris, and the orientation matrix is derived. Finally, the digital ortho map (DOM) and digital elevation model (DEM) of one orbit are generated automatically. The results show that the relative and absolute accuracies of triangulation are relatively high and generally acceptable. The visualised 3D realistic scene and the mosaicked DOM and DEM of lunar surface validate the feasibility of the proposed solution.

Robust surface matching by integrating edge segments

Abstract. This paper describes a novel area-based stereo-matching method which aims at reconstructing the shape of objects robustly, correctly, with high precision and with high density. Our goal is to reconstruct correctly the shape of the object by comprising also edges as part of the resulting surface. For this purpose, we need to overcome the problem of how to reconstruct and describe shapes with steep and sharp edges. Area-based matching methods set an image area as a template and search the corresponding match. As a direct consequence of this approach, it becomes not possible to correctly reconstruct the shape around steep edges. Moreover, in the same regions, discontinuities and discrepancies of the shape between the left and right stereo-images increase the difficulties for the matching process. In order to overcome these problems, we propose in this paper the approach of reconstructing the shape of objects by embedding reliable edge line segments into the area-based matching process with parallax estimation. We propose a robust stereo-matching (the extended Edge TIN-LSM) method which integrates edges and which is able to cope with differences in right and left image shape, brightness changes and occlusions. The method consists of the following three steps: (1) parallax estimation, (2) edge-matching, (3) edge-surface matching. In this paper, we describe and explain in detail the process of parallax estimation and the area-based surface-matching with integrated edges; the performance of the proposed method is also validated. The main advantage of this new method is its ability to reconstruct with high precision a 3D model of an object from only two images (for ex. measurement of a tire with 0.14 mm accuracy), thus without the need of a large number of images. For this reason, this approach is intrinsically simple and high-speed.

Evaluation of an Area-Based matching algorithm with advanced shape models

Abstract. Nowadays, the scientific institutions involved in planetary mapping are working on new strategies to produce accurate high resolution DTMs from space images at planetary scale, usually dealing with extremely large data volumes. From a methodological point of view, despite the introduction of a series of new algorithms for image matching (e.g. the Semi Global Matching) that yield superior results (especially because they produce usually smooth and continuous surfaces) with lower processing times, the preference in this field still goes to well established area-based matching techniques. Many efforts are consequently directed to improve each phase of the photogrammetric process, from image pre-processing to DTM interpolation. In this context, the Dense Matcher software (DM) developed at the University of Parma has been recently optimized to cope with very high resolution images provided by the most recent missions (LROC NAC and HiRISE) focusing the efforts mainly to the improvement of the correlation phase and the process automation. Important changes have been made to the correlation algorithm, still maintaining its high performance in terms of precision and accuracy, by implementing an advanced version of the Least Squares Matching (LSM) algorithm. In particular, an iterative algorithm has been developed to adapt the geometric transformation in image resampling using different shape functions as originally proposed by other authors in different applications.

3-Dimentional Mapping Coastal Zone using High Resolution Satellite Stereo Imageries

The metropolitan coastal zone mapping is critical for coastal resource management, coastal environmental protection, and coastal sustainable development and planning. The results of geometric processing of a Shanghai coastal zone from 0.7-m-resolution QuickBird Geo stereo images are presented firstly. The geo-positioning accuracy of ground point determination with vendor-provided rigorous physical model (RPM) parameters is evaluated and systematic errors are found when compared with ground control points surveyed by GPS real-time kinematic (GPS-RTK) with 5cm accuracy. A bias-compensation process in image space that applies a RPM bundle adjustment to the RPM-calculated 3D ground points to correct the systematic errors is used to improve the geo-positioning accuracy. And then, a area-based matching (ABM) method is used to generated the densely corresponding points of left and right QuickBird images. With the densely matching points, the 3-dimentinal coordinates of ground points can be calculated by using the refined geometric relationship between image and ground points. At last step, digital surface model (DSM) can be achieved automatically using interpolation method. Accuracies of the DSM as assessed from independent checkpoints (ICPs) are approximately 1.2 m in height.

MULTI-SCALE MATCHING FOR THE AUTOMATIC LOCATION OF CONTROL POINTS IN LARGE SCALE AERIAL IMAGES USING TERRESTRIAL SCENES

Abstract. A technique to automatically locate Ground Control Points (GCPs) in large aerial images is presented considering the availability of low accuracy direct georeferencing data. The approach is based on image chips of GCPs extracted from vertical terrestrial images. A strategy combining image matching techniques was implemented to select correct matches. These matches were used to define a 2D transformation with which the GCP is projected close to its correct position, reducing the search space in the aerial image. Area-based matching with some refinements is used to locate GCPs with sub-pixel precision. Experiments were performed with multi-scale images and assessed with a bundle block adjustment simulating an indirect sensor orientation. The accuracy analysis was accomplished based on discrepancies obtained from GCPs and check points. The results were better than interactive measurements and a planimetric accuracy of 1/5 of the Ground Sample Distance (GSD) for the check points was achieved.

21013 領域ベースマッチングを用いた画像計測における局所画像の特性評価

Area-based matching method is applied in applications of image processing in many fields. This method requires the block matching on the pixel unit to calculate the similarity of two local region images. The best similarity and surrounding better similarities are approximated by the quadric surface function for estimating the sub-pixel displacement. Although these method has the high resolution power, the accuracy is altered according to the local region image. Therefore, we propose a new evaluating method for categorizing the local region image. This method uses the self-similarity and the characterization of approximated quadric surface. In addition, we conduct the verification experiment on real captured image for confirmation of availability of proposed method.

Improved Seamless Cloning based Image In painting

Image in-painting is a restoration of image parts which are missing either accidentally or intentionally, and also image recovery is a research hot spot in the computer graphics and computer vision in the current. Two major inpainting techniques are introduced, those are PDE-based and Texture based algorithms. In PDE-based technique a conventional image matching techniques may be classified as either area based or feature based methods. In this paper, taking an input image and apply Pre-processing, After we get gray image and contour extraction algorithm is used to extract the contour features of damaged area, which determine the contour feature location. After that Matching algorithms play a key role in deciding correspondences between two image scenes. Area-based matching algorithm is used to search alternative matching area .Contour-based Similarity Distance function (CSD) is used to determine the most similar matching area. According to the damaged area and the matching area, the source cloning domains and target cloning domains of Seamless Cloning algorithm is improved. Using mean value coordinates to achieve regional pixel cloning from the source cloning domain to the target cloning domain. Finally, the improved SC algorithm is used to repair the image. Experimental results confirm the effectiveness of the proposed algorithm, and that the method can not only repair small-scale scratches, but also repair a large area damaged areas.