Initial Retrieval Results Research Articles

Graph Convolution Based Efficient Re-Ranking for Visual Retrieval

Visual retrieval tasks such as image retrieval and person re-identification (Re-ID) aim at effectively and thoroughly searching images with similar content or the same identity. After obtaining retrieved examples, re-ranking is a widely adopted post-processing step to reorder and improve the initial retrieval results by making use of the contextual information from semantically neighboring samples. Prevailing re-ranking approaches update distance metrics and mostly rely on inefficient crosscheck set comparison operations while computing expanded neighbors based distances. In this work, we present an efficient re-ranking method which refines initial retrieval results by updating features. Specifically, we reformulate re-ranking based on Graph Convolution Networks (GCN) and propose a novel Graph Convolution based Re-ranking (GCR) for visual retrieval tasks via feature propagation. To accelerate computation for large-scale retrieval, a decentralized and synchronous feature propagation algorithm which supports parallel or distributed computing is introduced. In particular, the plain GCR is extended for cross-camera retrieval and an improved feature propagation formulation is presented to leverage affinity relationships across different cameras. It is also extended for video-based retrieval, and Graph Convolution based Re-ranking for Video (GCRV) is proposed by mathematically deriving a novel profile vector generation method for the tracklet. Without bells and whistles, the proposed approaches achieve state-of-the-art performances on seven benchmark datasets from three different tasks, i.e., image retrieval, person Re-ID and video-based person Re-ID. Code is publicly available: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">github.com/WesleyZhang1991/GCN rerank</uri> .

Recallable Question Answering-Based Re-Ranking Considering Semantic Region for Cross-Modal Retrieval

Question answering (QA)-based re-ranking methods for cross-modal retrieval have been recently proposed to further narrow down similar candidate images. The conventional QA-based re-ranking methods provide questions to users by analyzing candidate images, and the initial retrieval results are re-ranked based on the user's feedback. Contrary to these developments, only focusing on performance improvement makes it difficult to efficiently elicit the user's retrieval intention. To realize more useful QA-based re-ranking, considering the user interaction for eliciting the user's retrieval intention is required. In this paper, we propose a QA-based re-ranking method with considering two important factors for eliciting the user's retrieval intention: query-image relevance and recallability. Considering the query-image relevance enables to only focus on the candidate images related to the provided query text, while, focusing on the recallability enables users to easily answer the provided question. With these procedures, our method can efficiently and effectively elicit the user's retrieval intention. Experimental results using Microsoft Common Objects in Context and computationally constructed dataset including similar candidate images show that our method can improve the performance of the cross-modal retrieval methods and the QA-based re-ranking methods.

A Novel Adaptive Feature Fusion Strategy for Image Retrieval.

In the era of big data, it is challenging to efficiently retrieve the required images from the vast amount of data. Therefore, a content-based image retrieval system is an important research direction to address this problem. Furthermore, a multi-feature-based image retrieval system can compensate for the shortage of a single feature to a certain extent, which is essential for improving retrieval system performance. Feature selection and feature fusion strategies are critical in the study of multi-feature fusion image retrieval. This paper proposes a multi-feature fusion image retrieval strategy with adaptive features based on information entropy theory. Firstly, we extract the image features, construct the distance function to calculate the similarity using the information entropy proposed in this paper, and obtain the initial retrieval results. Then, we obtain the precision of single feature retrieval based on the correlation feedback as the retrieval trust and use the retrieval trust to select the effective features automatically. After that, we initialize the weights of selected features using the average weights, construct the probability transfer matrix, and use the PageRank algorithm to update the initialized feature weights to obtain the final weights. Finally, we calculate the comprehensive similarity based on the final weights and output the detection results. This has two advantages: (1) the proposed strategy uses multiple features for image retrieval, which has better performance and more substantial generalization than the retrieval strategy based on a single feature; (2) compared with the fixed-feature retrieval strategy, our method selects the best features for fusion in each query, which takes full advantages of each feature. The experimental results show that our proposed method outperforms other methods. In the datasets of Corel1k, UC Merced Land-Use, and RSSCN7, the top10 retrieval precision is 99.55%, 88.02%, and 88.28%, respectively. In the Holidays dataset, the mean average precision (mAP) was 92.46%.

Sketch-Based Image Retrieval With Multi-Clustering Re-Ranking

To improve the performance of sketch-based image retrieval (SBIR) methods, most existing SBIR methods develop brand new SBIR methods. In fact, a re-ranking approach, which can refine the retrieval results of SBIR methods, is also beneficial. Inspired by this, in this paper, an SBIR re-ranking approach based on multi-clustering is proposed. In order to make the re-ranking approach invisible to users and adaptive to different types of image datasets, we made it an unsupervised method using blind feedback. Distinguished from the existing methods, this re-ranking approach uses the semantic information of three types of images: edge maps, object images (images with black background and natural images' foreground objects) and natural images themselves. With the initial retrieval results of an SBIR method, our approach first does the clustering operation for three types of images. Then, we utilize the clustering results to generate a cluster score for each initial retrieval result. Finally, the cluster score is used to calculate the final retrieval scores for the initial retrieval results. The experiments on different SBIR datasets are conducted. Experimental results demonstrate that, by implementing our re-ranking approach, the retrieval accuracy of a variety of SBIR methods is increased. Furthermore, the comparisons between our re-ranking method and the existing re-ranking methods are given.

Enhancing Sketch-Based Image Retrieval by CNN Semantic Re-ranking.

This paper introduces a convolutional neural network (CNN) semantic re-ranking system to enhance the performance of sketch-based image retrieval (SBIR). Distinguished from the existing approaches, the proposed system can leverage category information brought by CNNs to support effective similarity measurement between the images. To achieve effective classification of query sketches and high-quality initial retrieval results, one CNN model is trained for classification of sketches, another for that of natural images. Through training dual CNN models, the semantic information of both the sketches and natural images is captured by deep learning. In order to measure the category similarity between images, a category similarity measurement method is proposed. Category information is then used for re-ranking. Re-ranking operation first infers the retrieval category of the query sketch and then uses the category similarity measurement to measure the category similarity between the query sketch and each initial retrieval result. Finally, the initial retrieval results are re-ranked. The experiments on different types of SBIR datasets demonstrate the effectiveness of the proposed re-ranking method. Comparisons with other re-ranking algorithms are also given to show the proposed method's superiority. Further, compared to the baseline systems, the proposed re-ranking approach achieves significantly higher precision in the top ten different SBIR methods and datasets.

Relative similarity preserving bitwise weights generated by an adaptive mechanism

Due to its high query speed and low storage cost, binary hashing has been widely used in approximate nearest neighbors (ANN) search. However, the binary bits are generally considered to be equal, which causes data points with different codes to share the same Hamming distance to the query sample. To solve the above distance measure ambiguity, bitwise weights methods were proposed. Unfortunately, in most of the existing methods, the bitwise weights and the binary codes are learnt separately in two stages, and their performances cannot be further improved. In this paper, to effectively address the above issues, we propose an adaptive mechanism that jointly generate the bitwise weights and the binary codes by preserving different types of similarity relationship. As a result, the binary codes are utilized to obtain the initial retrieval results, and they are further re-ranked by the weighted Hamming distance. This ANN search mechanism is termed AR-Rank in this paper. First, this joint mechanism allows the bitwise weights and the binary codes to be used as mutual feedback during the training stage, and they are well adapted to one other when the algorithm converges. Furthermore, the bitwise weights are required to preserve the relative similarity which is consistent with the nature of ANN search task. Thus, the data points can be accurately re-sorted based on the weighted Hamming distances. Evaluations on three datasets demonstrate that the proposed AR-Rank retrieval system outperforms nine state-of-the-art methods.

A New Re-Ranking Method Based on Convolutional Neural Network and Two Image-to-Class Distances for Remote Sensing Image Retrieval

With the rapid growth of remote sensing image data, it has become necessary to effectively and efficiently retrieve images from a big image database for managing and exploiting such data. This paper presents a novel method for content-based remote sensing image retrieval (CBRSIR) that re-ranks the initial retrieval result using two image-to-class distances, which are the similarity between an image and an image class. One is the image-to-training-class distance between an image and each image class of the training dataset, which uses the information included in the train samples with known-label. It is used to calculate the weight of the image classes and estimating the class of an image. The other is the image-to-query-class distance between an image and the query class that is obtained using the k-nearest neighbor (k NN) method. The proposed method first obtains the initial retrieval results via sorting the distances of the CNN feature between the query image and each retrieved image in ascending order. Then, the image-to-query-class distance is calculated according to the initial result, and the weight of each class is calculated according to the class probability of images obtained by CNN and the image-to-training-class distances. Finally, the initial result is re-ranked to get the final retrieval result via the weighted image-to-class distances. The performance of the proposed method is tested on UCMD and PatternNet databases. Experimental results show the effectiveness of the proposed re-ranking scheme in image retrieval in comparison to other state-of-the-art techniques.

Intensity Filtering and Group Fusion for Accurate Mobile Place Recognition

Mobile place recognition targets at matching query images captured by mobile devices with database images collected from vehicle-mounted cameras, such as Google street view panoramas, which plays an important role in many applications. However, current solutions deriving from image retrieval suffer from the problem of low precision on top results, which significantly challenges their usability. By investigating the state-of-the-art approaches, we find that the bad illumination significantly affects initial results, and these initial results are correlative in both spatial location and visual content, which can be utilized for further improvement. In this paper, we propose an effective approach to rerank initial top-ranked results to improve the recognition recall. First, initial retrieval results with low intensity are filtered as they usually depict irrelevant places with dark background. Second, the correlation between top-ranked results is modeled as a reciprocal neighborhood graph by jointly considering spatial location and visual similarity. With the graph, the initial results are reranked based on voting similarity from the query and reciprocal neighbors. In this way, the underlying structure of initial retrieval results is exploited for refining. Experimental results on the public Tokyo 24/7 and San Francisco landmark datasets demonstrate that the proposed approach can achieve persisting improvement of recognition recall over the state-of-the-art approach.

Two-Stage Reranking for Remote Sensing Image Retrieval

Image reranking is a popular postprocessing method for remote sensing image retrieval (RSIR), which aims at enhancing the initial retrieval performance. In general, it takes either users’ opinions or the relationships between images into consideration to find an optimal reranked list based on the initial retrieved results. In this paper, we present a reranking method for improving RSIR, which is named two-stage reranking (TSR). Suppose the ${k}$ -nearest neighbors of a query RS image have been obtained by the initial retrieval. The first step of our TSR is to edit these neighbors using the editing scheme. A handful of informative and representative RS images are selected by the active learning algorithm, and their binary labels are provided by the users relative to the query image. Then, a binary classifier is trained using the selected RS images and their labels to classify the rest of the neighbors. Finally, both classification results and rank information in the initial retrieval results are considered to decide which neighbor should be excluded. In the next step, the remaining RS images are reranked by the proposed reranking scheme, i.e., multisimilarity fusion reranking. Both the user’s experience and image relationships are taken into account in TSR to ensure the performance of the reranking. The efficiency and the robustness of our method are validated by experiments conducted on two different types of RS images. Compared with the existing visual reranking approaches, our method achieves improved performance.

SAR Image Content Retrieval Based on Fuzzy Similarity and Relevance Feedback

This paper presents a new content-based synthetic aperture radar (SAR) image retrieval method to search out SAR image patches, which consists of two essential parts: an initial retrieval and later refined results. To obtain the proper initial retrievals, we develop a similarity measure named region-based fuzzy matching (RFM) to evaluate the similarities between SAR image patches. First, to reduce the negative influence of speckle noise, we segment the SAR image patches into brightness-texture regions at the superpixel level rather than the pixel level. Second, a multiscale edge detector is utilized to resolve the multiscale property of the SAR image patches, and then the edge regions of the SAR image patches are defined by those edge features. Third, to overcome the segmented uncertainty and the blurry boundaries, the obtained regions are described by fuzzy features. Finally, the RFM similarity between two SAR image patches is converted into the linear summation of the resemblance between different fuzzy feature sets. After we obtain the initial retrieval results, the multiple relevance feedback (MRF) scheme is proposed to refine the original ranked list. In this scheme, different relevance feedback approaches are carried out respectively, and then their results are fused to improve the initial retrieval. In addition, a new kernel function based on the RFM measure is developed for MRF. The encouraging experimental results counted on a manually constructed ground truth SAR image patch dataset demonstrate that our retrieval method is effective for SAR images compared with some existing approaches proposed in the remote sensing community

Fusion Similarity-Based Reranking for SAR Image Retrieval

A new reranking method, fusion similarity-based reranking, is proposed in this letter to improve the performance of synthetic aperture radar (SAR) image retrieval. First, the top ranked SAR images within the initial retrieval results are picked for reranking. Considering the negative influence of the speckle noise, three SAR-oriented visual features are selected to represent them. In addition, the different relevance scores corresponding to an SAR image are estimated in various modalities (i.e., different feature spaces). Second, a fusion similarity is defined under the relevance score space to measure the resemblance between two SAR images. This fusion similarity is calculated using the modal-image matrix, which is construed by the estimated scores to integrate the contributions of all modalities. Finally, an existing reranking function is adopted to rerank the SAR images with the help of the estimated scores and calculated fusion similarities. The positive experimental results demonstrate that our reranking method is effective and efficient.

VLAD re-ranking: Iteratively estimating the probability of relevance with relationships between dataset images

Vector of Locally Aggregated Descriptor (VLAD) and Bag-of-word (BOW) are two typical frameworks for specific object retrieval. Many enhancements of baseline BOW approach combine with the spatial information of local features, and can significantly improve the retrieval accuracy via spatial-information-based re-ranking approaches. The VLAD approach is designed as a method with high efficiency in terms of time and memory consumption, which accumulates local features as a compacted low-dimensional descriptor for image representation. Due to the fundamental difference between the two frameworks, methods for improving the accuracy of BOW cannot generally be applied for VLAD. In this paper, we introduce a retrieval framework that aims to combine the advantages of both VLAD and BOW. In the off-line part, image representations of both BOW and VLAD are generated, and we further use BOW to construct an image graph, which is a structure containing relationships between the dataset images. For real-time retrieval, we first follow the typical VLAD framework to obtain initial retrieval results. Then, an iterative re-ranking method is proposed to re-estimate the probabilities of relevance with the support of the image graph, and the final ranks of the images are obtained according to the values of probability values. The experimental results show that, compared to state-of-the-art VLAD methods, our framework achieves better retrieval accuracy with almost no compromise in time and memory efficiency.

Improving the effectiveness of language modeling approaches to information retrieval

Improving the effectiveness of language modeling approaches to information retrieval

Temperatures in the upper mesosphere and lower thermosphere from OSIRIS observations of O2 A-band emission spectra

Temperatures in the mesosphere – lower thermosphere region (MLT) have been derived from the Optical Spectrograph and InfraRed Imaging System (OSIRIS) observations of O2 A-band (b1Σ g+ –X3Σ g– )O–O emission spectra. The observed OSIRIS spectra are inverted pixel by pixel, producing inverted volume emission rate spectra at altitudes between 90 and 110 km, which are compared to modelled temperature dependent O2 A-band spectra. The estimated accuracy of the retrieved temperatures is approximately ±2 K near 90 km and up to ±6 K at higher altitudes. The developed temperature retrieval technique is presented, and some initial retrieval results are briefly discussed.

CBIR of spine X-ray images on inter-vertebral disc space and shape profiles using feature ranking and voting consensus

Very limited research is published in the literature that applies content-based image retrieval (CBIR) techniques to retrieval of digitized spine X-ray images that combines inter-vertebral disc space and vertebral shape profiles. This paper describes a novel technique for retrieving vertebra pairs that exhibit a specified disc space narrowing (DSN) and inter-vertebral disc shape. DSN is characterized using spatial and geometrical features between two adjacent vertebrae. In order to obtain the best retrieval result, all selected features are ranked and assigned a weight to indicate their importance in the computation of the final similarity measure. Using a two phase algorithm, initial retrieval results are clustered and used to construct a voting committee to retrieve vertebra pairs with the highest DSN similarity. The overall retrieval accuracy is validated by a radiologist and proves that selected features combined with voting consensus are effective for DSN-based spine X-ray image retrieval.