Image Geolocation Research Articles

An efficient cross-view image fusion method based on selected state space and hashing for promoting urban perception

In the field of cross-view image geolocation, traditional convolutional neural network (CNN)-based learning models generate unsatisfactory fusion performance due to their inability to model global correlations. The Transformer-based fusion methods can well compensate for the above problems, however, the Transformer has quadratic computational complexity and huge GPU memory consumption. The recent Mamba model based on the selection state space has a strong ability to model long sequences, lower GPU memory occupancy, and fewer GFLOPs. It is thus attractive and worth studying to apply Mamba to the cross-view image geolocation task. In addition, in the image-matching process (i.e., fusion of satellite/aerial and street view data.), we found that the storage occupancy of similarity measures based on floating-point features is high. Efficiently converting floating-point features into hash codes is a possible solution. In this study, we propose a cross-view image geolocation method (S6HG) based purely on Vision Mamba and hashing. S6HG fully utilizes the advantages of Vision Mamba in global information modeling and explicit location information encoding and the low storage occupancy of hash codes. Our method consists of two stages. In the first stage, we use a Siamese network based purely on vision Mamba to embed features for street view images and satellite images respectively. Our first-stage model is called S6G. In the second stage, we construct a cross-view autoencoder to further refine and compress the embedded features, and then simply map the refined features to hash codes. Comprehensive experiments show that S6G has achieved superior results on the CVACT dataset and comparable results to the most advanced methods on the CVUSA dataset. It is worth noting that other floating-point feature-based methods (4096-dimension) are 170.59 times faster than S6HG (768-bit) in storing 90,618 retrieval gallery data. Furthermore, the inference efficiency of S6G is higher than ViT-based computational methods.

A Satellite-Drone Image Cross-View Geolocalization Method Based on Multi-Scale Information and Dual-Channel Attention Mechanism

Satellite-Drone Image Cross-View Geolocalization has wide applications. Due to the pronounced variations in the visual features of 3D objects under different angles, Satellite-Drone cross-view image geolocalization remains an unresolved challenge. The key to successful cross-view geolocalization lies in extracting crucial spatial structure information across different scales in the image. Recent studies improve image matching accuracy by introducing an attention mechanism to establish global associations among local features. However, existing methods primarily focus on using single-scale features and employ a single-channel attention mechanism to correlate local convolutional features from different locations. This approach inadequately explores and utilizes multi-scale spatial structure information within the image, particularly lacking in the extraction and utilization of locally valuable information. In this paper, we propose a cross-view image geolocalization method based on multi-scale information and a dual-channel attention mechanism. The multi-scale information includes features extracted from different scales using various convolutional slices, and it extensively utilizes shallow network features. The dual-channel attention mechanism, through successive local and global feature associations, effectively learns depth discriminative features across different scales. Experimental results were conducted using existing satellite and drone image datasets, with additional validation performed on an independent self-made dataset. The findings indicate that our approach exhibits superior performance compared to existing methods. The methodology presented in this paper exhibits enhanced capabilities, especially in the exploitation of multi-scale spatial structure information and the extraction of locally valuable information.

Unified and Real-Time Image Geo-Localization via Fine-Grained Overlap Estimation.

Image geo-localization aims to locate a query image from source platform (e.g., drones, street vehicle) by matching it with Geo-tagged reference images from the target platforms (e.g., different satellites). Achieving cross-modal or cross-view real-time (>30fps) image localization with the guaranteed accuracy in a unified framework remains a challenge due to the huge differences in modalities and views between the two platforms. In order to solve this problem, a novel fine-grained overlap estimation based image geo-localization method is proposed in this paper, the core of which is to estimate the salient and subtle overlapping regions in image pairs to ensure correct matching. Specifically, the high-level semantic features of input images are extracted by a deep convolutional neural network. Then, a novel overlap scanning module (OSM) is presented to mine the long-range spatial and channel dependencies of semantic features in various subspaces, thereby identifying fine-grained overlapping regions. Finally, we adopt the triplet ranking loss to guide the proposed network optimization so that the matching regions are as close as possible and the most mismatched regions are as far away as possible. To demonstrate the effectiveness of our FOENet, comprehensive experiments are conducted on three cross-view benchmarks and one cross-modal benchmark. Our FOENet yields better performance in various metrics and the recall accuracy at top 1 (R@1) is significantly improved, with a maximum improvement of 70.6%. In addition, the proposed model runs fast on a single RTX 6000, reaching real-time inference speed on all datasets, with the fastest being 82.3 FPS.

Review of cross-view image geolocalization methods

Review of cross-view image geolocalization methods

CIG2S: A Cross-View Image Geo-Localization Model Based on G2S Transform Suitable for Center-Misaligned Scenarios

CIG2S: A Cross-View Image Geo-Localization Model Based on G2S Transform Suitable for Center-Misaligned Scenarios

4SCIG: A Four-Branch Framework to Reduce the Interference of Sky Area in Cross-View Image Geo-Localization

4SCIG: A Four-Branch Framework to Reduce the Interference of Sky Area in Cross-View Image Geo-Localization

UAV LARGE OBLIQUE IMAGE GEO-LOCALIZATION USING SATELLITE IMAGES IN THE DENSE BUILDINGS AREA

Abstract. For UAV large oblique image geo-localization in the dense buildings area, there are still two main challenges. One is the presence of obvious occlusion and large viewpoint differences in UAV images, and the other arises from the fact that reference images, particularly orthographic satellite images, lack façade information of man-made structures (such as buildings and roads), which is crucial for UAV large oblique images. Most of existing image-based geo-localization methods only address the first challenge, neglecting the interference brought by the second challenge, especially for UAV large oblique image geo-localization in the dense buildings area. Motivated by both these two challenges, we have proposed a novel method for UAV large oblique image geo-localization in the dense buildings areas, with the segments direction statistics (SDS) features and their histogram descriptors designed. By considering both the local and global features of man-made structures, the proposed method effectively addresses the significant information difference encountered in cross-view image matching. We conducted experiments on both the public UAV images dataset University-1652 and our own collected dataset of UAV large oblique long focal whiskbroom (LO-LF-W) images. Comparative analysis with state-of-the-art (SOTA) methods demonstrated that the proposed method improves the geo-localization accuracy by approximately 10%. Furthermore, the proposed method exhibits greater robustness to noise and changing orientation of reference images, making it particularly well-suited for dense buildings areas that pose challenges for existing methods.

Integrating Multi-Point Geostatistics, Machine Learning, and Image Correlation for Characterizing Positional Errors in Remote-Sensing Images of High Spatial Resolution

Remote-sensing images of high spatial resolution (HSR) are valuable sources of fine-grained spatial information for various applications, such as urban surveys and governance. There is continuing research on positional errors in remote-sensing images and their impacts in geoprocessing and applications. This paper explores the combined use of multi-point geostatistics (MPS), machine learning—in particular, generalized additive modeling (GAM)—and computer-image correlation for characterizing positional errors in images—in particular, HSR images. These methods are employed because of the merits of MPS in being flexible for non-parametric and joint simulation of positional errors in X and Y coordinates, the merits of GAM in being capable of handling non-stationarity in-positional errors through error de-trending, and the merits of computer-image correlation in being cost-effective in furnishing the training data (TD) required in MPS. Procedurally, image correlation is applied to identify homologous image points in reference-test image pairs to extract image displacements automatically in constructing TD. To cope with the complexity of urban scenes and the unavailability of truly orthorectified images, visual screening is performed to clean the raw displacement data to create quality-enhanced TD, while manual digitization is used to obtain reference sample data, including conditioning data (CD), for MPS and test data for performance evaluation. GAM is used to decompose CD and TD into trends and residuals. With CD and TD both de-trended, the direct sampling (DS) algorithm for MPS is applied to simulate residuals over a simulation grid (SG) at 80 m spatial resolution. With the realizations of residuals and, hence, positional errors generated in this way, the means, standard deviation, and cross correlation in bivariate positional errors at SG nodes are computed. The simulated error fields are also used to generate equal-probable realizations of vertices that define some road centerlines (RCLs), selected for this research through interpolation over the aforementioned simulated error fields, leading to error metrics for the RCLs and for the lengths of some RCL segments. The enhanced georectification of the RCLs is facilitated through error correction. A case study based in Shanghai municipality, China, was carried out, using HSR images as part of generalized point clouds that were developed. The experiment results confirmed that by using the proposed methods, spatially explicit positional-error metrics, including means, standard deviation, and cross correlation, can be quantified flexibly, with those in the selected RCLs and the lengths of some RCL segments derived easily through error propagation. The reference positions of these RCLs were obtained through error correction. The positional accuracy gains achieved by the proposed methods were found to be comparable with those achieved by conventional image georectification, in which the CD were used as image-georectification control data. The proposed methods are valuable not only for uncertainty-informed image geolocation and analysis, but also for integrated geoinformation processing.

Semantic-guided de-attention with sharpened triplet marginal loss for visual place recognition

Thanks to Earth-level Street View images from Google Maps, a visual image geo-localization can estimate the coarse location of a query image with a visual place recognition process. However, this can get very challenging when non-static objects change with time, severely degrading image retrieval accuracy. We address the problem of city-scale visual place recognition in complex urban environments crowded with non-static clutters. To this end, we first analyze what clutters degrade similarity matching between the query and database images. Second, we design a self-supervised trainable de-attention module that prevents the network from focusing on non-static objects in an input image. In addition, we propose a novel triplet marginal loss called sharpened triplet marginal loss to make feature descriptors more discriminative. Lastly, due to the lack of geo-tagged public datasets with a high density of non-static objects, we propose a clutter augmentation method to evaluate our approach. The experimental results show that our model has notably improved over the existing attention methods in geo-localization tasks on the public benchmark datasets and on their augmented versions with high population and traffic. Our code is available at https://github.com/ccsmm78/deattention_with_stml_for_vpr.

AENet: attention efficient network for cross-view image geo-localization

<abstract><p>To address the problem that task-irrelevant objects such as cars, pedestrians and sky, will interfere with the extracted feature descriptors in cross-view image geo-localization, this paper proposes a novel method for cross-view image geo-localization, named as AENet. The method includes two main parts: an attention efficient network fusing channel and spatial attention mechanisms and a triplet loss function based on a multiple hard samples weighting strategy. In the first part, the EfficientNetV2 network is used to extract features from the images and preliminarily filter irrelevant features from the channel dimension, then the Triplet Attention layer is applied to further filter irrelevant features from the spatial dimension. In the second part, a multiple hard samples weighting strategy is proposed to enhance the learning of hard samples. Experimental results show that our proposed method significantly outperforms the state-of-the-art method on two existing benchmark datasets.</p></abstract>

Context-Aware Querying, Geolocalization, and Rephotography of Historical Newspaper Images

Newspapers contain a wealth of historical information in the form of articles and illustrations. Libraries and cultural heritage institutions have been digitizing their collections for decades to enable web-based access to and retrieval of information. A number of challenges arise when dealing with digitized collections, such as those of KBR, the Royal Library of Brussels (used in this study), which contain only page-level metadata, making it difficult to extract information from specific contexts. A context-aware search relies heavily on metadata enhancement. Therefore, when using metadata at the page level, it is even more challenging to geolocalize less-known landmarks. To overcome this challenge, we have developed a pipeline for geolocalization and visualization of historical photographs. The first step of this pipeline consists of converting page-level metadata to article-level metadata. In the next step, all articles with building images were classified based on image classification algorithms. Moreover, to correctly geolocalize historical photographs, we propose a hybrid approach that uses both textual metadata and image features. We conclude this research paper by addressing the challenge of visualizing historical content in a way that adds value to humanities research. It is noteworthy that a number of historical urban scenes are visualized using rephotography, which is notoriously challenging to get right. This study serves as an important step towards enriching historical metadata and facilitating cross-collection linkages, geolocalization, and the visualization of historical newspaper images. Furthermore, the proposed methodology is generic and can be used to process untagged photographs from social media, including Flickr and Instagram.

Improving Street View Image Classification Using Pre-trained CNN Model Extracted Features

This paper presents a new approach for the challenging problem of image geo-localization using Convolutional Neural Networks (CNNs). This latter has become the state-of-the-art technique in computer vision and machine learning, particularly in location recognition of images taken in urban environments where the recognition accuracy is very impressive. We cast this task as a classification problem. First, we extract features from images by using pre-trained CNN model AlexNet as a feature extraction tool; where the output of the fully connected layer is considered as the feature representation. Then, the features extracted from the fully connected layer can be used for the classification process by feeding them into the Support Vector Machine (SVM) classifier. We evaluated the proposed approach on a data set of Google Street View images (GSV); the experimental results show that our approach can improve the classification by achieving a good accuracy rate which is 94.19%.

AN OPEN API FOR 3D-GEOREFERENCED HISTORICAL PICTURES

Abstract. In this article we describe the possibilities and opportunities of the utilization of 3D georeferenced historical images in different contexts. In the frame of an ongoing research project that aims at establishing a national infrastructure for georeferenced pictures, we have created an open API that allows researchers and developers to access and interact with an existing database containing 200’000 3D-georeferenced images. In this database images have been georeferenced by volunteers in 3D using monoplotting. The new API allows for instance to extract the 3D geolocation of images and their footprints as well as to upload new images. It is as far as possible based on existing standards in the fields of archive management, 3D image standards and web-services. This work is of interest for researchers who want to utilize and analyze 3D georeferenced historical imagery and for people who want to establish open API’s to give access to data that is relevant for research.

Image Geolocation Method Based on Attention Mechanism Front Loading and Feature Fusion

Image geolocation is an important technique for robotics and autonomous systems. The existing methods mainly extract local features from images directly and use global descriptors, which are aggregated by these local features, to retrieve candidate references from all references. Thus, the training efficiency is affected by the image noises and the accuracy is so limited that the further verification is extremely time consuming. To address these issues, this work proposes an image geolocation framework, which adds the noise filtering layer before local feature extraction. Based on this framework, an image geolocation method based on attention mechanism front loading and feature fusion is designed. In the noise filtering layer, the proposed method uses triplet attention to denoise images thus leading to higher training efficiency. In the feature aggregation layer, an improved SPP (spatial pyramid pooling) is designed to extract the local factors reflected by the position relationships among local features. Then, the local factors are incorporated with the global factors extracted by NetVLAD. The fused descriptors contain not only the statistic of the geometric elements but also the position relationships among them. The experimental results show that the proposed method outperforms NetVLAD in terms of the training convergence round and Recall@ N ( N = 1 , 5 , 10 , 20 ); especially, the convergence round of Recall@5 reduces from 25 to 10, the convergence round of Recall@10 reduces from 25 to 7, Recall@1 increases from 79.45% to 84.01%, and Recall@5 increases from 90.10% to 92.81%.

Learning Cross-Scale Visual Representations for Real-Time Image Geo-Localization

Robot localization remains a challenging task in GPS denied environments. State estimation approaches based on local sensors, e.g. cameras or IMUs, are drifting-prone for long-range missions as error accumulates. In this study, we aim to address this problem by localizing image observations in a 2D multi-modal geospatial map. We introduce the cross-scale dataset and a methodology to produce additional data from cross-modality sources. We propose a framework that learns cross-scale visual representations without supervision. Experiments are conducted on data from two different domains, underwater and aerial. In contrast to existing studies in cross-view image geo-localization, our approach a) performs better on smaller-scale multi-modal maps; b) is more computationally efficient for real-time applications; c) can serve directly in concert with state estimation pipelines.

Cross-view Image Geo-Localization using Multi-Scale Generalized Pooling with Attention Mechanism

Cross-view Image Geo-Localization using Multi-Scale Generalized Pooling with Attention Mechanism

Deep Learning-Based Image Geolocation for Travel Recommendation via Multi-Task Learning

Localizing images by visual information is a very challenging task in image-based travel recommendations. Travelers take a large number of pictures every day and share them on social networks (Facebook, Sina Weibo, Yelp, etc.). Many of these images are associated with the location where they are taken. But for images that do not associate with geographic location information, how to estimate where they are taken? With the rapid development of social media, the increasing number of shared geographic-labeled images brings an opportunity to address this problem. Using geographic-labeled images to estimate the location of unlabeled images is a popular approach. In this paper, we propose an image geographic location estimation model via multi-task learning (GLML). It combines the classification task and retrieval task to calculate the similarity between the query image and dataset images. Additionally, it fuses multi-global features through multiple global pooling techniques to enhance feature extraction. Each part of the proposed GLML model is flexible and extensible. Experiments on seven public datasets show the effectiveness of the proposed model.

Soft Exemplar Highlighting for Cross-View Image-Based Geo-Localization.

The goal of ground-to-aerial image geo-localization is to determine the location of a ground query image by matching it against a reference database consisting of aerial/satellite images. This task is highly challenging due to the large appearance difference caused by extreme changes in viewpoint and orientation. In this work, we show that the training difficulty is an important cue that can be leveraged to improve metric learning on cross-view images. More specifically, we propose a new Soft Exemplar Highlighting (SEH) loss to achieve online soft selection of exemplars. Adaptive weights are generated for exemplars by measuring their associated training difficulty using distance rectified logistic regression. These weights are then constrained to remove simple exemplars from training and truncate the large weights of extremely hard exemplars to escape from the trap with a local optimal solution. We further use the proposed SEH loss to train two mainstream convolutional neural networks for ground-to-aerial image-based geo-localization. Experimental results on two benchmark cross-view image datasets demonstrate that the proposed method achieves significant improvements in feature discriminativeness and outperforms the state-of-the-art image-based geo-localization methods.

Accurate 3-DoF Camera Geo-Localization via Ground-to-Satellite Image Matching.

We address the problem of ground-to-satellite image geo-localization, that is, estimating the camera latitude, longitude and orientation (azimuth angle) by matching a query image captured at the ground level against a large-scale database with geotagged satellite images. Our prior arts treat the above task as pure image retrieval by selecting the most similar satellite reference image matching the ground-level query image. However, such an approach often produces coarse location estimates because the geotag of the retrieved satellite image only corresponds to the image center while the ground camera can be located at any point within the image. To further consolidate our prior research finding, we present a novel geometry-aware geo-localization method. Our new method is able to achieve the fine-grained location of a query image, up to pixel size precision of the satellite image, once its coarse location and orientation have been determined. Moreover, we propose a new geometry-aware image retrieval pipeline to improve the coarse localization accuracy. Apart from a polar transform in our conference work, this new pipeline also maps satellite image pixels to the ground-level plane in the ground-view via a geometry-constrained projective transform to emphasize informative regions, such as road structures, for cross-view geo-localization. Extensive quantitative and qualitative experiments demonstrate the effectiveness of our newly proposed framework. We also significantly improve the performance of coarse localization results compared to the state-of-the-art in terms of location recalls.

Location of Synthetic Aperture Radar Imagery via Range History

Geolocation is one of the key synthetic aperture radar (SAR) image processing procedures in SAR-based remote sensing applications. The conventional SAR imagery geolocation model is based on the range-Doppler (RD) equation, which is derived from the Fresnel approximation of SAR. Currently, there is no other rigorous geolocation model for modern high-resolution SAR; however, this approximation may not be fully applicable for various modern SAR imaging modes and algorithms. This study aims to develop a novel SAR image geolocation model based on the SAR range history (RH) equation, which is derived from the basic SAR backscattered signal property. Moreover, a new geometric calibration model is presented for RH-based location. Experiments based on real spaceborne SAR data are presented as examples, and the location accuracy of RH and RD location models are compared. The results show that the location accuracies of the RH and RD models are consistent under the existing SAR processing framework, but those for the RH model demonstrate better performance in the azimuth direction. In theory, the proposed RH model can be used for SAR image products generated from complex modes and non-Fresnel-approximation-based imaging algorithms, making it suitable for future SAR systems.