Remote Sensing Image Retrieval Research Articles

Mental Retrieval of Remote Sensing Images via Adversarial Sketch-Image Feature Learning

Searching the targets of interest in large-scale remote sensing images is a fundamental problem, which becomes a very challenging issue when there is no relevant example at hand but a mental picture in mind. Hand-drawn sketch as a precise and convenient expression of the mental picture makes sketch-based remote sensing image retrieval (SBRSIR) an ideal choice to cope with this issue. However, the accuracy of SBRSIR algorithm, which is critical to effective retrieval, is still far behind classic query-by-example image retrieval. Two central limiting factors for this performance gap are: 1) the lack of effective cross-domain representations for bridging the domain gap between sketches and remote sensing images and 2) the absence of large sketch/remote sensing image data sets for developing, evaluating, and comparing the SBRSIR approaches. In this article, we first develop a novel SBRSIR model to learn a deep joint embedding space with discriminative losses, where adversarial training is used for the embedding space to learn domain-invariant representations. Then, we contribute a sketch/remote sensing image data set specifically for SBRSIR and provide a benchmark for subsequent researchers. Extensive experiments on the data set and large scene images demonstrate the effectiveness and superiority of the method for both the seen and unseen categories.

Slice-feature based deep hashing algorithm for remote sensing image retrieval

Remote sensing image retrieval is aimed at obtaining a target image from a typically large number of remote sensing images. Different from natural scene images, remote sensing images usually includes more channels such as near-infrared reflectances (NIR), which will result in higher data dimensionality. However, high dimensionality of image features may lead to information redundancy and high computational costs. Recently, deep hashing methods to reduce image feature dimensionality have been proposed. However, compared with natural images, remote sensing images have smaller interclass distances, which hinder the application of deep hashing methods for image retrieval. To overcome this problem, in this paper, we develop a slice-feature deep hashing (SFDH) method for remote sensing image retrieval. The major contribution of this work is proposing an image correlation reduction strategy by separating the features in the fully connected layers into many slices. The SFDH architecture is based on a pre-trained Inception V3 model; the proposed model not only extracts image features from remote sensing images, but also introduces the slice-feature strategy to improve the fully connected structure of the previously proposed metric learning based deep hashing network (MiLaN). Besides, triplet loss is utilized to further enlarge differences between classes. Our experimental results demonstrate that the proposed slice-feature strategy outperforms state-of-the-art remote sensing image retrieval methods.

Hashing Nets for Hashing: A Quantized Deep Learning to Hash Framework for Remote Sensing Image Retrieval

Fast and accurate remote sensing image retrieval from large data archives has been an important research topic in the remote sensing research literature. Recently, hashing-based remote sensing image retrieval has attracted extreme attention because of its efficient search capabilities. Especially, deep remote sensing image hashing algorithms have been developed based on convolutional neural networks (CNNs) and have shown effective retrieval performance. However, implementing a deep hashing network tends to be highly expensive in terms of storage space and computing resources to be suitable for on-orbit remote sensing image retrieval, which usually operates on resource-limited devices such as satellites and unmanned aerial vehicles (UAVs). To address this limitation, we propose to hash a deep network that in turn hashes remote sensing images. Specifically, we develop a quantized deep learning to hash (QDLH) framework for large-scale remote sensing image retrieval. The weights and activation functions in the QDLH framework are binarized to low-bit representations, which require comparatively much less storage space and computing resources. The QDLH results in a lightweight deep neural network for effective remote sensing image hashing. We conduct extensive experiments on two public remote sensing image data sets by incorporating several state-of-the-art network architectures into our QDLH methodology for remote sensing image hashing. The experimental results demonstrate that the proposed QDLH is effective in saving hardware resources in terms of both storage and computation. Moreover, superior remote sensing image retrieval performance is also achieved by our QDLH, compared with state-of-the-art deep remote sensing image hashing methods.

Low-Rank Hypergraph Hashing for Large-Scale Remote Sensing Image Retrieval

As remote sensing (RS) images increase dramatically, the demand for remote sensing image retrieval (RSIR) is growing, and has received more and more attention. The characteristics of RS images, e.g., large volume, diversity and high complexity, make RSIR more challenging in terms of speed and accuracy. To reduce the retrieval complexity of RSIR, a hashing technique has been widely used for RSIR, mapping high-dimensional data into a low-dimensional Hamming space while preserving the similarity structure of data. In order to improve hashing performance, we propose a new hash learning method, named low-rank hypergraph hashing (LHH), to accomplish for the large-scale RSIR task. First, LHH employs a l2-1 norm to constrain the projection matrix to reduce the noise and redundancy among features. In addition, low-rankness is also imposed on the projection matrix to exploit its global structure. Second, LHH uses hypergraphs to capture the high-order relationship among data, and is very suitable to explore the complex structure of RS images. Finally, an iterative algorithm is developed to generate high-quality hash codes and efficiently solve the proposed optimization problem with a theoretical convergence guarantee. Extensive experiments are conducted on three RS image datasets and one natural image dataset that are publicly available. The experimental results demonstrate that the proposed LHH outperforms the existing hashing learning in RSIR tasks.

A Content based Remote Sensing Image Retrieval Using A Modified Local Color Descriptors

A Content based Remote Sensing Image Retrieval Using A Modified Local Color Descriptors

Multi-view Deep Representations with Cross-Dataset Transfer for Remote Sensing Image Retrieval and Classification

Transfer learning is a challenging task in computer vision, due to the differences of data distribution. Although convolutional neural network (CNN) could learn different levels of image abstraction, the single-view features extracted from the final layer of a pre-trained or fine-tuned CNN may result in insufficient image description over different datasets. To address this issue, we focus on deep representations with multi-view analysis for remote sensing image (RSI) retrieval and classification tasks using five recently released large-scale datasets. First, cross-dataset transfer learning is presented by fine-tuning a pre-trained CNN on one dataset and testing the fine-tuned network on another one. Second, multi-view image representations are explored in terms of different activation vectors as well as CNNs. Finally, a multi-view fusion and a random projection (RP) strategy are proposed to improve the accuracies and computational cost of both RSI tasks, respectively. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.

A Residual-Dyad Encoder Discriminator Network for Remote Sensing Image Matching

The advent of deep perceptual networks brought about a paradigm shift in machine vision and image perception. Image apprehension lately carried out by hand-crafted features in the latent space have been replaced by deep features acquired from supervised networks for improved understanding. However, such deep networks require strict supervision with a substantial amount of the labeled data for authentic training process. These methods perform poorly in domains lacking labeled data especially in case of remote sensing image retrieval. Resolving this, we propose an unsupervised encoder-decoder feature for remote sensing image matching (RSIM). Moreover, we replace the conventional distance metrics with a deep discriminator network to identify the similarity of the image pairs. To the best of our knowledge, discriminator network has never been used before for solving RSIM problem. Results have been validated with two publicly available benchmark remote sensing image datasets. The technique has also been investigated for content-based remote sensing image retrieval (CBRSIR); one of the widely used applications of RSIM. Results demonstrate that our technique supersedes the state-of-the-art methods used for unsupervised image matching with mean average precision (mAP) of 81%, and image retrieval with an overall improvement in mAP score of about 12%.

A Progressive Content-Based Image Retrieval in JPEG 2000 Compressed Remote Sensing Archives

Due to the dramatically increased volume of remote sensing (RS) image archives, images are usually stored in a compressed format to reduce the storage size. Existing content-based RS image retrieval (CBIR) systems require as input fully decoded images, thus resulting in a computationally demanding task in the case of large-scale CBIR problems. To overcome this limitation, in this article, we present a novel CBIR system that achieves a coarse-to-fine progressive RS image description and retrieval in the partially decoded Joint Photographic Experts Group (JPEG) 2000 compressed domain. The proposed system initially: 1) decodes the code blocks associated only to the coarse wavelet resolution and 2) discards the most irrelevant images to the query image based on the similarities computed on the coarse resolution wavelet features of the query and archive images. Then, the code blocks associated with the subsequent resolution of the remaining images are decoded and the most irrelevant images are discarded by computing similarities considering the image features associated with both resolutions. This is achieved by using the pyramid match kernel similarity measure that assigns higher weights to the features associated with the finer wavelet resolution than to those related to the coarse wavelet resolution. These processes are iterated until the codestreams associated with the highest wavelet resolution are decoded. Then, the final retrieval is performed on a very small set of completely decoded images. Experimental results obtained on two benchmark archives of aerial images point out that the proposed system is much faster while providing a similar retrieval accuracy than the standard CBIR systems.

Learning to Translate for Cross-Source Remote Sensing Image Retrieval

Content-based remote sensing image retrieval (CBRSIR) is one of the important techniques for the mining and analysis of big remote sensing data. Recently, unified-source CBRSIR (US-CBRSIR) has been extensively studied and explored, but there has been little attention on cross-source CBRSIR (CS-CBRSIR). Although there is motivation for exploration of CS-CBRSIR for the continually increasing multisource remote sensing data, CS-CBRSIR suffers data drift due to multisource data. In this article, we explore to explicitly address the problem by mapping the source domain to target domain and propose an image translation-based framework for CS-CBRSIR. On the one hand, a novel cycle-identity-generative adversarial network (CI-GAN) is proposed based on the cycle-GAN. In addition to the generator and discriminator, a pretrained classifier, identity module, is designed to further boost the discriminative ability of translated images and facilitate the implementation of feature extraction and similarity measure. On the other hand, to alleviate the impact of style difference between the generated and real images, translated image augmentations and label smoothing regularization (LSR) are adopted to enhance training and contribute toward generation of a robust feature extractor. Extensive experiments on a public data set and a detailed ablation study confirm the effectiveness of our approach.

Similarity Retention Loss (SRL) Based on Deep Metric Learning for Remote Sensing Image Retrieval

Recently, with the rapid growth of the number of datasets with remote sensing images, it is urgent to propose an effective image retrieval method to manage and use such image data. In this paper, we propose a deep metric learning strategy based on Similarity Retention Loss (SRL) for content-based remote sensing image retrieval. We have improved the current metric learning methods from the following aspects—sample mining, network model structure and metric loss function. On the basis of redefining the hard samples and easy samples, we mine the positive and negative samples according to the size and spatial distribution of the dataset classes. At the same time, Similarity Retention Loss is proposed and the ratio of easy samples to hard samples in the class is used to assign dynamic weights to the hard samples selected in the experiment to learn the sample structure characteristics within the class. For negative samples, different weights are set based on the spatial distribution of the surrounding samples to maintain the consistency of similar structures among classes. Finally, we conduct a large number of comprehensive experiments on two remote sensing datasets with the fine-tuning network. The experiment results show that the method used in this paper achieves the state-of-the-art performance.

Coarse-to-Fine Deep Metric Learning for Remote Sensing Image Retrieval

Remote sensing image retrieval (RSIR) is the process of searching for identical areas by investigating the similarities between a query image and the database images. RSIR is a challenging task owing to the time difference, viewpoint, and coverage area depending on the shooting circumstance, resulting in variations in the image contents. In this paper, we propose a novel method based on a coarse-to-fine strategy, which makes a deep network more robust to the variations in remote sensing images. Moreover, we propose a new triangular loss function to consider the whole relation within the tuple. This loss function improves the retrieval performance and demonstrates better performance in terms of learning the detailed information in complex remote sensing images. To verify our methods, we experimented with the Google Earth South Korea dataset, which contains 40,000 images, using the evaluation metric Recall@n. In all experiments, we obtained better performance results than those of the existing retrieval training methods. Our source code and Google Earth South Korea dataset are available online.

Global Optimal Structured Embedding Learning for Remote Sensing Image Retrieval.

A rich line of works focus on designing elegant loss functions under the deep metric learning (DML) paradigm to learn a discriminative embedding space for remote sensing image retrieval (RSIR). Essentially, such embedding space could efficiently distinguish deep feature descriptors. So far, most existing losses used in RSIR are based on triplets, which have disadvantages of local optimization, slow convergence and insufficient use of similarity structure in a mini-batch. In this paper, we present a novel DML method named as global optimal structured loss to deal with the limitation of triplet loss. To be specific, we use a softmax function rather than a hinge function in our novel loss to realize global optimization. In addition, we present a novel optimal structured loss, which globally learn an efficient deep embedding space with mined informative sample pairs to force the positive pairs within a limitation and push the negative ones far away from a given boundary. We have conducted extensive experiments on four public remote sensing datasets and the results show that the proposed global optimal structured loss with pairs mining scheme achieves the state-of-the-art performance compared with the baselines.

Distribution Consistency Loss for Large-Scale Remote Sensing Image Retrieval

Remote sensing images are featured by massiveness, diversity and complexity. These features put forward higher requirements for the speed and accuracy of remote sensing image retrieval. The extraction method plays a key role in retrieving remote sensing images. Deep metric learning (DML) captures the semantic similarity information between data points by learning embedding in vector space. However, due to the uneven distribution of sample data in remote sensing image datasets, the pair-based loss currently used in DML is not suitable. To improve this, we propose a novel distribution consistency loss to solve this problem. First, we define a new way to mine samples by selecting five in-class hard samples and five inter-class hard samples to form an informative set. This method can make the network extract more useful information in a short time. Secondly, in order to avoid inaccurate feature extraction due to sample imbalance, we assign dynamic weight to the positive samples according to the ratio of the number of hard samples and easy samples in the class, and name the loss caused by the positive sample as the sample balance loss. We combine the sample balance of the positive samples with the ranking consistency of the negative samples to form our distribution consistency loss. Finally, we built an end-to-end fine-tuning network suitable for remote sensing image retrieval. We display comprehensive experimental results drawing on three remote sensing image datasets that are publicly available and show that our method achieves the state-of-the-art performance.

Toward Remote Sensing Image Retrieval Under a Deep Image Captioning Perspective

The performance of remote sensing image retrieval (RSIR) systems depends on the capability of the extracted features in characterizing the semantic content of images. Existing RSIR systems describe images by visual descriptors that model the primitives (such as different land-cover classes) present in the images. However, the visual descriptors may not be sufficient to describe the high-level complex content of RS images (e.g., attributes and relationships among different land-cover classes). To address this issue, in this article, we present an RSIR system that aims at generating and exploiting textual descriptions to accurately describe the relationships between the objects and their attributes present in RS images with captions (i.e., sentences). To this end, the proposed retrieval system consists of three main steps. The first step aims to encode the image visual features and then translate the encoded features into a textual description that summarizes the content of the image with captions. This is achieved based on the combination of a convolutional neural network with a recurrent neural network. The second step aims to convert the generated textual descriptions into semantically meaningful feature vectors. This is achieved by using the recent word embedding techniques. Finally, the last step estimates the similarity between the vectors of the textual descriptions of the query image and those of the archive images, and then retrieve the most similar images to the query image. Experimental results obtained on two different datasets show that the description of the image content with captions in the framework of RSIR leads to an accurate retrieval performance.

Multilabel Remote Sensing Image Retrieval Based on Fully Convolutional Network

Conventional remote sensing image retrieval (RSIR) systems usually perform single-label retrieval where each image is annotated by a single label representing the most significant semantic content of the image. This assumption, however, ignores the complexity of remote sensing images, where an image might have multiple classes (i.e., multiple labels), thus resulting in worse retrieval performance. We therefore propose a novel multi-label RSIR approach with fully convolutional networks (FCN). In our approach, we first train a FCN model using a pixel-wise labeled dataset,and the trained FCN is then used to predict the segmentation maps of each image in the considered archive. We finally extract region convolutional features of each image based on its segmentation map.The region features can be either used to perform region-based retrieval or further post-processed to obtain a feature vector for similarity measure. The experimental results show that our approach achieves state-of-the-art performance in contrast to conventional single-label and recent multi-label RSIR approaches.

Query-Adaptive Remote Sensing Image Retrieval Based on Image Rank Similarity and Image-to-Query Class Similarity

Many image features have been proposed for image retrieval; hence, effectively fusing these features to alleviate the large variation in performance among image queries when using single image features has become a major challenge in remote sensing (RS) image retrieval. Because high-resolution remote sensing images have abundant and complex visual contents, accurately measuring the similarity between two images is another important problem. To address these challenges, we propose a novel RS image retrieval method that uses query-adaptive feature weights to fuse features and utilizes two image similarities to improve retrieval performance. First, we use the image rank similarity, which measures the similarity between two images according to their corresponding top-m image lists from a reference image collection, to calculate the similarity of each feature between a query image and each retrieved image. Then, we assign a weight to each feature to fuse these features via our query-adaptive weighting method. Finally, we take the query image and its neighborhood set selected from the retrieval dataset as the query class and utilize the image-to-query class similarity to re-rank the retrieval results. Extensive experiments are conducted on two publicly available RS image databases. Compared with the state-of-the-art methods, the proposed method can significantly enhance the retrieval precision.

An Application of Ternary Hash Retrieval Method for Remote Sensing Images in Panoramic Video

With the development of multimedia technology, the application of panoramic video is emerging, and there are specific application scenarios in the field of drones. Remote sensing images have tremendous applications in data acquisition and processing in the panoramic maps used to construct the panoramic videos. Efficient remote sensing image retrieval can greatly improve the efficiency of image utilization and get a fast response in the construction of panoramic video. Thus, how to achieve fast and efficient remote sensing image retrieval becomes increasingly important. Scholars have explored a number of ways to address this problem, from traditional manual feature-based retrieval methods to deep hash retrieval methods that today can cope with large data volumes. The hash method is a way to encode high-dimensional data into a set of binary codes while maintaining similarity between images. In this paper, we propose an end-to-end ternary label depth hashing method that is capable of both feature learning and hash coding processes in one-stage. The experimental results show that the algorithm has high accuracy and good retrieval effect.

A Deep Cross-Modality Hashing Network for SAR and Optical Remote Sensing Images Retrieval

The content-based remote sensing image retrieval (CBRSIR) has recently become a hot topic due to its wide applications in analysis of remote sensing data. However, since conventional CBRSIR is unsuitable in harsh environments, this article focuses on the cross-modality CBRSIR (CM-CBRSIR) between synthetic aperture radar (SAR) and optical images. Besides the large interclass and small intraclass in CBRSIR, CM-CBRSIR is limited by prominent modality discrepancy caused by different imaging mechanisms. To address this limitation, this study proposes a deep cross-modality hashing network. First, we transform optical images with three channels into four different types of single-channel images to increase diversity of the training modalities. This helps the network to mainly focus on extracting the contour and texture shared features and makes it less sensitive to color information for images across modalities. Second, we combine any type of randomly selected transformed images and its corresponding SAR or optical images to form image pairs that are fed into the networks. The training strategy, with paired image data, eliminates the large cross-modality variations caused by different modalities. Finally, the triplet loss, in combination with the hash function, helps the modal to extract the discriminative features of images and upgrade the retrieval efficiency. To further evaluate the proposed modality, we construct a SAR-optical dual-modality remote sensing image dataset containing 12 categories. Experimental results demonstrate the superiority of the proposed method with regards to efficiency and generality.

A Discriminative Distillation Network for Cross-Source Remote Sensing Image Retrieval

Nowadays, several remote sensing image capturing technologies are used ranging from unmanned aerial vehicles to satellites. Powerful learning-based discriminative features play an essential role in content-based remote sensing image retrieval (CBRSIR). Cross-source CBRSIR (CS-CBRSIR) is used to find relevant remote sensing images across different remote sensing sources (i.e., multispectral images and panchromatic images). But it is limited by large cross-source and intrasource variations caused by different semantic objects, spatial resolution, and spectral resolution. The main limitation of CS-CBRSIR is that it cannot address the inconsistency between different sources and exploit the intrinsic relation between them. This study proposes a discriminative distillation network for CS-CBRSIR to address this limitation. To enlarge the interclass variations and reduce the intraclass differences, the discriminative features from the first source are first extracted with a well-designed joint optimization configuration (JOC) on the basis of deep neural networks. Thereafter, the features extracted from the first source are used as a supervision signal for the second source; feature distribution in common feature space between the first and second sources are made significantly similar. The method proposed in this study simultaneously handles the cross-source and intersource variations, unlike the existing methods. Extensive experiments on the DSRSID dataset with Euclidean distance verify the effectiveness of our proposed method.

基于区域注意力机制的遥感图像检索

基于区域注意力机制的遥感图像检索