Feature Matching Step Research Articles

Solving photogrammetric cold cases using AI-based image matching: New potential for monitoring the past with historical aerial images

With the ongoing digitization in archives, an increasing number of historical data becomes available for research. This includes historical aerial images which provide detailed information about the depicted area. Among the applications enabled by these images are change detection of land use, land cover, glaciers, and coastal environments as well as the observation of land degradation, and natural hazards. Studying the depicted areas and occurring 3D deformations requires the generation of a digital surface model (DSM) which is usually obtained via photogrammetric Structure-from-Motion (SfM). However, conventional SfM workflows often fail in registering historical aerial images due to their radiometric characteristics introduced by digitization, original image quality, or vast temporal changes between epochs. We demonstrate that the feature matching step in the Structure from Motion (SfM) pipeline is particularly crucial. To address this issue, we apply the two synergetic neural network methods SuperGlue and DISK, improving feature matching for historical aerial images. This requires several modifications to enable rotational invariance and leveraging the high resolution of aerial images. In contrast to other studies our workflow does not require any prior information such as DSMs, flight height, focal lengths, or scan resolution which are often no more extent in archives. It is shown that our methods using adapted parameter settings are even able to deal with quasi texture-less images. This enables the simultaneous processing of various kind of mono-temporal and multi-temporal data handled in a single workflow from data preparation over feature matching through to camera parameter estimation and the generation of a sparse point cloud. It outperforms conventional strategies in the number of correct feature matches, number of registered images and calculated 3D points and allows the generation of multi-temporal DSMs with high quality.With the flexibility of the method, it enables the automatic processing of formerly unusable or only to be interactively processed data, e.g. aerial images where the flight route is unknown, or with difficult radiometric properties. This makes it possible to go back even further in time, where the data quality usually decreases, and enables a holistic monitoring and comparison of environments of high interest. The code is made publicly available at https://github.com/tudipffmgt/HAI-SFM.

Deep learning-based fast volumetric imaging using kV and MV projection images for lung cancer radiotherapy: A feasibility study.

The long acquisition time of CBCT discourages repeat verification imaging, therefore increasing treatment uncertainty. In this study, we present a fast volumetric imaging method for lung cancer radiation therapy using an orthogonal 2D kV/MV image pair. The proposed model is a combination of 2D and 3D networks. The proposed model consists of five major parts: (1) kV and MV feature extractors are used to extract deep features from the perpendicular kV and MV projections. (2) The feature-matching step is used to re-align the feature maps to their projection angle in a Cartesian coordinate system. By using a residual module, the feature map can focus more on the difference between the estimated and ground truth images. (3) In addition, the feature map is downsized to include more global semantic information for the 3D estimation, which is useful to reduce inhomogeneity. By using convolution-based reweighting, the model is able to further increase the uniformity of image. (4) To reduce the blurry noise of generated 3D volume, the Laplacian latent space loss calculated via the feature map that is extracted via specifically-learned Gaussian kernel is used to supervise the network. (5) Finally, the 3D volume is derived from the trained model. We conducted a proof-of-concept study using 50 patients with lung cancer. An orthogonal kV/MV pair was generated by ray tracing through CT of each phase in a 4D CT scan. Orthogonal kV/MV pairs from nine respiratory phases were used to train this patient-specific model while the kV/MV pair of the remaining phase was held for model testing. The results are based on simulation data and phantom results from a real Linac system. The mean absolute error (MAE) values achieved by our method were 57.5 HU and 77.4 HU within body and tumor region-of-interest (ROI), respectively. The mean achieved peak-signal-to-noise ratios (PSNR) were 27.6dB and 19.2dB within the body and tumor ROI, respectively. The achieved mean normalized cross correlation (NCC) values were 0.97 and 0.94 within the body and tumor ROI, respectively. A phantom study demonstrated that the proposed method can accurately re-position the phantom after shift. It is also shown that the proposed method using both kV and MV is superior to current method using kV or MV only in image quality. These results demonstrate the feasibility and accuracy of our proposed fast volumetric imaging method from an orthogonal kV/MV pair, which provides a potential solution for daily treatment setup and verification of patients receiving radiation therapy for lung cancer.

Satellite image matching and registration using affine transformation and hybrid feature descriptors

Image registration (IR) is an image processing technique to determine geometrical transformation that gives the most accurate match between reference and floating images. In existing IR work, the inliers ratio and outlier's ratios are equal. This reduces the image registration accuracy. So, to avoid the outlier's ratio and increase the inliers ratio, hybrid invariant local features descriptor is proposed. This feature descriptor consists of binary-robust-invariant-scalable-key point (BRISK), speed-up-robust-feature (SURF), and feature from accelerated segment test feature (FAST). The hybrid feature descriptor extracts the relevant features from the image. Then, the feature matching step finds the correct correspondences from the two sets of features. Also, affine transformation avoids the false feature matching points. An experimental analysis shows the inliers ratio and repeatability evaluation metrics performance of individual feature descriptor, combined feature descriptor and proposed feature descriptor. The proposed hybrid feature descriptor achieves inliers ratio of 1.913 and repeatability is 0.121.

A Robust Edge-Preserving Stereo Matching Method for Laparoscopic Images.

Stereo matching has become an active area of research in the field of computer vision. In minimally invasive surgery, stereo matching provides depth information to surgeons, with the potential to increase the safety of surgical procedures, particularly those performed laparoscopically. Many stereo matching methods have been reported to perform well for natural images, but for images acquired during a laparoscopic procedure, they are limited by image characteristics including illumination differences, weak texture content, specular highlights, and occlusions. To overcome these limitations, we propose a robust edge-preserving stereo matching method for laparoscopic images, comprising an efficient sparse-dense feature matching step, left and right image illumination equalization, and refined disparity optimization. We validated the proposed method using both benchmark biological phantoms and surgical stereoscopic data. Experimental results illustrated that, in the presence of heavy illumination differences between image pairs, texture and textureless surfaces, specular highlights and occlusions, our proposed approach consistently obtains a more accurate estimate of the disparity map than state-of-the-art stereo matching methods in terms of robustness and boundary preservation.

POTENTIAL EXPLORATION OF SEGMENTATION DRIVEN STEREO MATCHING OF VERY HIGH-RESOLUTION SATELLITE IMAGERY

Abstract. The automatic establishment of image relationship between oblique images can be a challenging task. Feature based image matching (FBM) establishes this relationship by detecting and matching corresponding feature points. A robust matching is beneficial for many tasks including reconstruction, mapping and localization. The need of automatic processing of remotely sensed data, like very highresolution (VHR) satellite imagery, increased over the years. Furthermore, green vegetation and water are changing the physical properties with respect to the amount of light emitted, season and pollution. In addition, with human interaction, the change in appearance increases. The Normalized Differential Vegetation Index (NDVI) is a well-known and studied index in order to detect healthy green vegetation. The Normalized Differential Water Index (NDWI) can help identify water areas in an image. They can be used to preliminarily segment images into different categories for later process. This study proposes a novel framework to explore the potential of feature based stereo matching for very high-resolution satellite imagery with segmentation. The proposed framework will perform first, image segmentation with NDVI and NDWI on stereo VHR satellite image pairs. Then, classification by threshold to detect healthy green vegetation, water and image frame. Features within these three classes are masked out and kept from being processed during the feature matching step. The idea is that, features in these classes are easy to cause miss matching because they are more prone to be affected by different image and environmental conditions in the stereo image pairs. As a result, the amount of miss matches can be reduced on average by 7.5%. Furthermore, the segmentation decreases the total amount of detected features by 13.71%, so that the processing time decreases. This study has successfully proven that segmentation can lead to improved stereo matching. In future, segmentation driven can be utilized by AI matching processes as well as traditionally photogrammetric or computer vision tasks.

Robust 3D Reconstruction Using HDR-Based SLAM

3D reconstruction is an important topic in the field of the emerging applications such as smart robotics, virtual reality (VR), augmented reality (AR), and autonomous driving. RGB-D simultaneous localization and mapping (SLAM) technique is widely used in the reconstruction process. However, low light and low textured environment often results in insufficient point features and fails the reconstruction. To address this problem, we propose a robust RGB-D SLAM system using high dynamic range (HDR) image information called HDR-based SLAM. The deep learning based HDR generation method is adopted to map a single low dynamic range (LDR) image into a radiance map which is normalized to exclude the influence of exposure time. We retrained the ORB descriptor patch to fit the normalized radiance maps in the feature matching step. The proposed method can improve the quantitative camera trajectory accuracy and qualitative result of geometry reconstruction. Experimental results show that the proposed method has better performance compared to that of the standard range imaging SLAM under challenging low light environment, which helps expand the applicability of 3D reconstruction system.

Shape retrieval by using multi‐scale angle‐based representation and dynamic label propagation

To improve the robustness and discrimination power of the triangle-area representation, a novel shape matching method based on multi-scale angle representation is proposed in this study. By analysing the configurations of different sample points from each shape contour, shape descriptors are constructed by using space angles at different scale levels. With the proposed shape representation, the multi-scale information of shape contours is efficiently described, and the dynamic programming is further used to determine the correspondence between samples from different shapes and calculate the shape distance in the feature matching step. Moreover, to improve the shape retrieval results based on pairwise shape distances, the dynamic label propagation is introduced as the post-processing step. Unlike previous distance learning methods learning the database manifold implicitly, the authors method retrieves relative objects on the shortest paths from near to far explicitly, and the underlying structure can be effectively captured. The proposed method tested on different shape databases provides the performances superior to many other methods, and it can be applied to visual data processing and understanding of the internet of things.

Brand Design of Coastal Eco-tourism Products Based on Cultural Creativity

Zhou, Y., 2020. Brand design of coastal eco-tourism products based on cultural creativity. In: Li, L. and Huang, X. (eds.), Sustainable Development in Coastal Regions: A Perspective of Environment, Economy, and Technology. Journal of Coastal Research, Special Issue No. 112, pp. 306-310. Coconut Creek (Florida), ISSN 0749-0208.At present, the brand design of marine tourism products does not have the characteristics of color bionics, and the visual information of the key target of packaging is not prominent, which leads to low interest of tourists in the products and poor sales of coastal eco-tourism products. Brand design method of coastal eco-tourism products based on cultural creativity is proposed. It designs the external functions of brand and combines with augmented reality technology to achieve the unity of packaging and virtual presentation. Color bionic feature extraction algorithm is used to enhance the visual effect of product packaging by block feature matching and other steps. By selecting the key frame image, the homography matrix between the current image frame and the key frame is calculated to obtain the absolute pose information of the current frame relative to the real world, so as to ensure the accurate matching between the real and the virtual. Finally, the experimental results show that the design effect of the proposed method achieves the purpose of the design scheme, which can significantly improve product sales.

Near-Duplicate Image Detection System Using Coarse-to-Fine Matching Scheme Based on Global and Local CNN Features

Due to the great success of convolutional neural networks (CNNs) in the area of computer vision, the existing methods tend to match the global or local CNN features between images for near-duplicate image detection. However, global CNN features are not robust enough to combat background clutter and partial occlusion, while local CNN features lead to high computational complexity in the step of feature matching. To achieve high efficiency while maintaining good accuracy, we propose a coarse-to-fine feature matching scheme using both global and local CNN features for real-time near-duplicate image detection. In the coarse matching stage, we implement the sum-pooling operation on convolutional feature maps (CFMs) to generate the global CNN features, and match these global CNN features between a given query image and database images to efficiently filter most of irrelevant images of the query. In the fine matching stage, the local CNN features are extracted by using maximum values of the CFMs and the saliency map generated by the graph-based visual saliency detection (GBVS) algorithm. These local CNN features are then matched between images to detect the near-duplicate versions of the query. Experimental results demonstrate that our proposed method not only achieves a real-time detection, but also provides higher accuracy than the state-of-the-art methods.

A novel method for digital image copy-move forgery detection and localization using evolving cellular automata and local binary patterns

Copy-Move Forgery Detection (CMFD) methods aim to forensically analyze a digital image for a possible content duplication manipulation. In the past, many block-based algorithms have been proposed for detection and localization of CMF. However, the existing solutions show limited efficacy for images compressed in JPEG and lack robustness against post-processing attacks such as noise addition, blurring, etc. To address this problem, we propose a new block-based passive method for detection and localization of CMF in this paper. Passive methods, as opposed to active methods, are used to authenticate the image content in the absence of any pre-embedded information such as watermarks. In our proposed scheme, a suspicious input image to be analyzed is first low pass filtered and converted to Local Binary Patterns (LBP) image. The LBP texture image is then divided into overlapping blocks. Next, a compact five-dimensional feature vector is extracted from each block by employing thresholding and Cellular Automata. The set of feature vectors is sorted lexicographically to bring the copy-pasted blocks nearer to each other. Finally, the feature matching step is used to reveal the duplicate blocks. Our experimental results indicate that the proposed method performs exceptionally well relative to other state-of-art-methods, under different image manipulation scenarios.

Implementation of Smart Attendance Monitoring Using Open-CV and Python

Uniqueness or individuality of a person is his face. In this paper, face of an individual is used for the purpose of attendance taking automatically. Maintaining attendance of a student is very important task in all the institutes to check the performance of students. Every institute has its own way for taking attendance. Some of the institutes use paper or file based approach, and some are using methods of automatic attendance taking using biometrics methods which is very time consuming process. There are many methods available for this purpose. Facial recognition may be a unambiguously distinctive or corroboratory an individual by comparison and analysing the patterns supported person’s facial contours. Facial recognition is mostly used for security purposes at so many places. This process can be divided into two phases, processing before recognition where face detection takes place, and afterwards face recognition takes place through feature extraction and matching steps. This system uses the face recognition for taking automatic attendance of students in the classroom without student’s intervention. This attendance is recorded by using a camera that captures images of students and detect the faces after which it compares and faces are detected with the database and mark the attendance for the students. Attendance sheet will be generated and message will be sent to the parents.

A 4D information system for the exploration of multitemporal images and maps using photogrammetry, web technologies and VR/AR

<p class="VARAbstract">This contribution shows the comparison, investigation, and implementation of different access strategies on multimodal data. The first part of the research is structured as a theoretical part opposing and explaining the terms of conventional access, virtual archival access, and virtual museums while additionally referencing related work. Especially, issues that still persist in repositories like the ambiguity or missing of metadata is pointed out. The second part explains the practical implementation of a workflow from a large image repository to various four-dimensional applications. Mainly, the filtering of images and in the following, the orientation of images is explained. Selection of the relevant images is partly done manually but also with the use of deep convolutional neural networks for image classification. In the following, photogrammetric methods are used for finding the relative orientation between image pairs in a projective frame. For this purpose, an adapted Structure from Motion (SfM) workflow is presented, in which the step of feature detection and matching is replaced by the Radiant-Invariant Feature Transform (RIFT) and Matching On Demand with View Synthesis (MODS). Both methods have been evaluated on a benchmark dataset and performed superior than other approaches. Subsequently, the oriented images are placed interactively and in the future automatically in a 4D browser application showing images, maps, and building models Further usage scenarios are presented in several Virtual Reality (VR) and Augmented Reality (AR) applications. The new representation of the archival data enables spatial and temporal browsing of repositories allowing the research of innovative perspectives and the uncovering of historical details.</p><p>Highlights:</p><ul><li>Strategies for a completely automated workflow from image repositories to four-dimensional (4D) access approaches.</li><li>The orientation of historical images using adapted and evaluated feature matching methods.</li><li>4D access methods for historical images and 3D models using web technologies and Virtual Reality (VR)/Augmented Reality (AR).</li></ul><p> </p>

Efficient Properties-Based Learning for Mismatch Removal

Mismatch removal is a critical step of feature matching, which is a prerequisite of many vision-based tasks. This paper aims to develop a general and robust method for mismatch removal. To this end, we propose an efficient learning-based mismatch removal method that can significantly improve outlier identification in terms of both accuracy and efficiency. The key idea of our approach is to use a set of properties to describe the putative matches and feed the match representations to a supervised learning procedure learning a binary classifier for mismatch removal. The efficient properties mainly include three aspects: consistency of neighborhood elements, weighted consistency of neighborhood topology, and the stability of correspondence. Different from existing consistency of neighborhood topology, we adopt a weighted strategy to emphasize the effect of different properties with respect to the identified correspondence. The match representations combine the spatial positions of the correspondences with their descriptor reliabilities, which can effectively enlarge the distributions between outliers and inliers. To handle large proportions of outliers, we design a simple strategy to obtain a subset with high ratio inliers guiding the match representations construction process. This strategy can also boost the number of true correspondences without sacrificing the accuracy. Extensive experiments demonstrate our superiority over the state-of-the-art methods.

Sea Ice Drift Tracking From Sequential SAR Images Using Accelerated-KAZE Features

In this paper, we propose a feature-tracking algorithm for sea ice drift retrieval from a pair of sequential satellite synthetic aperture radar (SAR) images. The method is based on feature tracking comprising feature detection, description, and matching steps. The approach exploits the benefits of nonlinear multiscale image representations using accelerated-KAZE (A-KAZE) features, a method that detects and describes image features in an anisotropic scale space. We evaluated several state-of-the-art feature-based algorithms, including A-KAZE, Scale Invariant Feature Transform (SIFT), and a very fast feature extractor that computes binary descriptors known as Oriented FAST and Rotated BRIEF (ORB) on dual polarized Sentinel-1A C-SAR extra wide swath mode data over the Arctic. The A-KAZE approach outperforms both ORB and SIFT up to an order of magnitude in ice drift. The experimental results showed high relevance of the proposed algorithm for retrieval of ice drift at subkilometre resolution from a pair of SAR images with 100-m pixel size. From this paper, we found that feature tracking using nonlinear scale-spaces is preferable due to its high efficiency against noise with respect to image features compared with other existing feature tracking alternatives that make use of Gaussian or linear scale spaces.

Hybrid Biometric System Using Iris and Speaker Recognition

In this study, a hybrid security system is proposed. The proposed system is composed of two subsystems namely iris recognition system (IRS) and speaker recognition system (SRS). Pre-processing, feature extraction and feature matching are the main steps of these systems. In IRS subsystem, Gaussian filter, Canny edge detector, Hough transform, and histogram equalization is performed for pre-processing, respectively. After that, by applying 4-level Discrete Wavelet Transform (DWT) to pure iris image, the iris image is decomposed into four sub-bands (LL4, LH4, HL4 and HH4). In order to extract the feature vector from iris pattern, the LH4, HL4 and HH4 sub-bands (matrices) are merged into one matrix. Finally the matrix is transformed in vector to obtain the feature vector of iris image. For SRS subsystem, the pre-processing step includes spectral arrangement, silence part removing and band limitation operations. After pre-processing, frame blocking and windowing are applied to the long-term speech samples and then Fast Fourier Transform (FFT) is performed for the each short-term speech segments (frames). Finally, the Mel Frequency Cepstral Coefficients (MFCC) technique is performed in order to obtain feature vector of the speech. The feature matching step of both IRS and SRS is implemented with Dynamic Time Warping (DTW) which is an efficient algorithm to measure the distance between two vectors. According to the DTW results, the false acceptance rate (FAR) is zero and false rejecting rate (FRR) is about 4 % for the proposed hybrid system.

Improved Feature Matching for Mobile Devices with IMU.

Thanks to the recent diffusion of low-cost high-resolution digital cameras and to the development of mostly automated procedures for image-based 3D reconstruction, the popularity of photogrammetry for environment surveys is constantly increasing in the last years. Automatic feature matching is an important step in order to successfully complete the photogrammetric 3D reconstruction: this step is the fundamental basis for the subsequent estimation of the geometry of the scene. This paper reconsiders the feature matching problem when dealing with smart mobile devices (e.g., when using the standard camera embedded in a smartphone as imaging sensor). More specifically, this paper aims at exploiting the information on camera movements provided by the inertial navigation system (INS) in order to make the feature matching step more robust and, possibly, computationally more efficient. First, a revised version of the affine scale-invariant feature transform (ASIFT) is considered: this version reduces the computational complexity of the original ASIFT, while still ensuring an increase of correct feature matches with respect to the SIFT. Furthermore, a new two-step procedure for the estimation of the essential matrix E (and the camera pose) is proposed in order to increase its estimation robustness and computational efficiency.

Qualified matching feature collection for feature point-based copy-move forgery detection

The feature matching step plays a critical role during the copy-move forgery detection procedure. However, when several highly similar features simultaneously exist in the feature space, current feature match- ing methods will miss a considerable number of genuine matching feature pairs. To this end, we propose a clustering-based method to collect qualified matching features for the feature point-based methods. The pro- posed method can collect far more genuine matching features than existing methods do, and thus significantly improve the detection performance, especially for multiple pasting cases. Experimental results confirm the effi- cacy of the proposed method. © 2015 SPIE and IS&T (DOI: 10.1117/1.JEI.24.2.023035)

Using an improved SIFT algorithm and fuzzy closed-loop control strategy for object recognition in cluttered scenes.

Partial occlusions, large pose variations, and extreme ambient illumination conditions generally cause the performance degradation of object recognition systems. Therefore, this paper presents a novel approach for fast and robust object recognition in cluttered scenes based on an improved scale invariant feature transform (SIFT) algorithm and a fuzzy closed-loop control method. First, a fast SIFT algorithm is proposed by classifying SIFT features into several clusters based on several attributes computed from the sub-orientation histogram (SOH), in the feature matching phase only features that share nearly the same corresponding attributes are compared. Second, a feature matching step is performed following a prioritized order based on the scale factor, which is calculated between the object image and the target object image, guaranteeing robust feature matching. Finally, a fuzzy closed-loop control strategy is applied to increase the accuracy of the object recognition and is essential for autonomous object manipulation process. Compared to the original SIFT algorithm for object recognition, the result of the proposed method shows that the number of SIFT features extracted from an object has a significant increase, and the computing speed of the object recognition processes increases by more than 40%. The experimental results confirmed that the proposed method performs effectively and accurately in cluttered scenes.

Multiscale Point Correspondence Using Feature Distribution and Frequency Domain Alignment

In this paper, a hybrid scheme is proposed to find the reliable point‐correspondences between two images, which combines the distribution of invariant spatial feature description and frequency domain alignment based on two‐stage coarse to fine refinement strategy. Firstly, the source and the target images are both down‐sampled by the image pyramid algorithm in a hierarchical multi‐scale way. The Fourier‐Mellin transform is applied to obtain the transformation parameters at the coarse level between the image pairs; then, the parameters can serve as the initial coarse guess, to guide the following feature matching step at the original scale, where the correspondences are restricted in a search window determined by the deformation between the reference image and the current image; Finally, a novel matching strategy is developed to reject the false matches by validating geometrical relationships between candidate matching points. By doing so, the alignment parameters are refined, which is more accurate and more flexible than a robust fitting technique. This in return can provide a more accurate result for feature correspondence. Experiments on real and synthetic image‐pairs show that our approach provides satisfactory feature matching performance.

Modifications in the SIFT operator for effective SAR image matching

With the increasing availability and rapidly improving the spatial resolution of synthetic aperture radar (SAR) images from the latest and future satellites like TerraSAR-X and TanDEM-X, their applicability in remote sensing applications is set to be paramount. Considering challenges in the field of point feature-based multisensor/multimodal SAR image matching/registration and advancements in the field of computer vision, we extend the applicability of the scale invariant feature transform (SIFT) operator for SAR images. In this article, we have analysed the feature detection, identification and matching steps of the original SIFT processing chain. We implement steps to counter the speckle influence, which deteriorates the SIFT operator performance for SAR images. In feature identification, we evaluate different local gradient estimating techniques and highlight the fact that giving up the SIFT's rotation invariance characteristic increases the potential number of matches when the multiple SAR images from different sensors have been acquired with the same geometrical acquisition parameters. In the feature matching stage, we propose to assist the standard SIFT matching scheme to utilise the SIFT operator capability for effective results in challenging SAR image matching scenarios. The results obtained for SAR images acquired by different sensors using different incidence angles and orbiting directions over both rural and semi urban land cover, highlight the SIFT operator's capability for point feature matching in SAR imagery.