Product Quantization Research Articles

3-1 近似最近傍探索のための直積量子化コードブック作成手法に関する一検討(第3部門符号化・復号化)

3-1 近似最近傍探索のための直積量子化コードブック作成手法に関する一検討(第3部門符号化・復号化)

Scalable Image Retrieval by Sparse Product Quantization

Fast Approximate Nearest Neighbor (ANN) search technique for high-dimensional feature indexing and retrieval is the crux of large-scale image retrieval. A recent promising technique is Product Quantization, which attempts to index high-dimensional image features by decomposing the feature space into a Cartesian product of low dimensional subspaces and quantizing each of them separately. Despite the promising results reported, their quantization approach follows the typical hard assignment of traditional quantization methods, which may result in large quantization errors and thus inferior search performance. Unlike the existing approaches, in this paper, we propose a novel approach called Sparse Product Quantization (SPQ) to encoding the high-dimensional feature vectors into sparse representation. We optimize the sparse representations of the feature vectors by minimizing their quantization errors, making the resulting representation is essentially close to the original data in practice. Experiments show that the proposed SPQ technique is not only able to compress data, but also an effective encoding technique. We obtain state-of-the-art results for ANN search on four public image datasets and the promising results of content-based image retrieval further validate the efficacy of our proposed method.

Online Variable Coding Length Product Quantization for Fast Nearest Neighbor Search in Mobile Retrieval

Quantization methods are crucial for efficient nearest neighbor search in many applications such as image, music, or product search. As mobile devices are becoming increasingly more popular, the quantization methods on mobile devices are more important, because a large portion of the search queries are becoming performed on mobile devices. One important characteristic of the communication on mobile devices is the inherent unreliability of their communication channels. In order to adapt the quality changes of the communication channels, we need to change the coding length of the quantization accordingly. The existing quantization methods use fixed-length codebooks, and it is expensive to retrain another codebook with different coding length. In this paper, we propose a novel variable length product quantization framework that consists of a set of fast universal scalar quantizers. The framework is capable of producing variable length quantization without retraining the codebook. Each data vector is transformed into a new space to reduce the correlation across dimensions. A proper number of bits is allocated to represent the scalar component in each dimension according to the given coding length. For each component, we estimate its probability density function (PDF) and design an efficient universal scalar quantizer based on the PDF and the allocated bits. To reduce distortion, we learn a Gaussian mixture model for the data. The experimental results show that, compared to state-of-the-art product quantization methods, our approach can construct the codebooks online for variable coding lengths and achieve the comparable performance.

Observables, Evolution Equation, and Stationary States Equation in the Joint Probability Representation of Quantum Mechanics

Symplectic and optical joint probability representations of quantum mechanics are considered, in which the functions describing the states are the probability distributions with all random arguments (except the argument of time ). The general formalism of quantizers and dequantizers determining the star product quantization scheme in these representations is given. Taking the Gaussian functions as the distributions of the tomographic parameters the correspondence rules for most interesting physical operators are found and the expressions of the dual symbols of operators in the form of singular and regular generalized functions are derived. Evolution equations and stationary states equations for symplectic and optical joint probability distributions are obtained.

Kernelized product quantization

There has been increasing interest in learning compact binary codes for large-scale image data representation and retrieval. In most existing hashing-based methods, high-dimensional vectors are hashed into Hamming space, and the similarity between two vectors is approximated by the Hamming distance between their binary codes. Although hashing-based binary codes generation methods were widely used, Product Quantization (PQ) has been shown to be more accurate than various hashing-based methods, largely due to its lower quantization distortions and more precise distance computation. However, it is still a challenging problem to generalize PQ to accommodate arbitrary kernels. In this paper, we demonstrate how to employ arbitrary kernel functions in a PQ scheme. First, we propose a Kernelized PQ (KPQ) method based on composite kernels, which serves as a basic framework by making the decomposition of implicit feature space possible. Furthermore, we propose a Kernelized Optimized PQ (KOPQ) method to generalize Optimized Product Quantization (OPQ) to an arbitrary implicit feature space. Finally, we propose a Supervised KPQ (SKPQ) to improve the performance of semantic neighbor search. Both methods are variations of KPQ with the incorporation of their corresponding core techniques, KPCA and KCCA respectively, to the basic KPQ framework. Experiments involving three notable datasets show that KPQ, KOPQ and SKPQ can outperform the state-of-the-art methods for a similarity search in feature space or semantic search.

Sketch-based manga retrieval using manga109 dataset

Manga (Japanese comics) are popular worldwide. However, current e-manga archives offer very limited search support, i.e., keyword-based search by title or author. To make the manga search experience more intuitive, efficient, and enjoyable, we propose a manga-specific image retrieval system. The proposed system consists of efficient margin labeling, edge orientation histogram feature description with screen tone removal, and approximate nearest-neighbor search using product quantization. For querying, the system provides a sketch-based interface. Based on the interface, two interactive reranking schemes are presented: relevance feedback and query retouch. For evaluation, we built a novel dataset of manga images, Manga109, which consists of 109 comic books of 21,142 pages drawn by professional manga artists. To the best of our knowledge, Manga109 is currently the biggest dataset of manga images available for research. Experimental results showed that the proposed framework is efficient and scalable (70 ms from 21,142 pages using a single computer with 204 MB RAM).

Dynamic programming based optimized product quantization for approximate nearest neighbor search

Product quantization and its variances have emerged in approximate nearest neighbor search, with a wide range of applications. However, the optimized division of product subspaces retains as an open problem that largely degenerates the retrieval accuracy. In the paper, an extremely optimized product quantization scheme is introduced, which ensures, both theoretically and experimentally, a much better subspace partition comparing to the existing state-of-the-arts PQ and OPQ. The key innovation is to formulate subspace partition as a graph-based optimization problem, by which dynamic programming is leveraged to pursuit optimal quantizer learning. Another advantage is that the proposed scheme is very easily integrated with the cutting-edge multi-indexing structure, with a nearly eligible overhead in addition. We have conducted a serial of large-scale quantitative evaluations, with comparisons to a group of recent works including PQ, OPQ, and multi-Index. We have shown superior performance gain in the widely used SIFT1B benchmark, which validates the merits of the proposed algorithm.

The distributed system for inverted multi-index visual retrieval

With the explosive growth of visual databases, it is infeasible to maintain the huge indexing structures within the memory of a single server. In this paper, a distributed visual retrieval system based on inverted multi-index is proposed to generate a huge codebook with very low memory consumption and time cost. In order to improve the performance of product quantization, a vector space decomposition strategy is performed by affinity propagation clustering. In the meantime, a distributed framework is introduced to inverted multi-index to improve the time efficiency. Our works are validated on the large scale database of INRIA Holidays and Flickr 1M. The results of our experiments indicate that the performance of PQ is greatly improved and the visual retrieval system is speeded up at comparable precision.

Parametric and nonparametric residual vector quantization optimizations for ANN search

For approximate nearest neighbor (ANN) search in many vision-based applications, vector quantization (VQ) is an efficient compact encoding technology. A representative approach of VQ is product quantization (PQ) which quantizes subspaces separately by Cartesian product and achieves high accuracy. But its space decomposition still leads to quantization distortion. This paper presents two optimized solutions based on residual vector quantization (RVQ). Different from PQ, RVQ simulates restoring quantization error by multi-stage quantizers instead of decomposing it. To further optimize codebook and space decomposition, we try to get a better discriminated space projection. Then an orthonormal matrix R is generated. The RVQ's nonparametric solution alternately optimizes R and stage-codebooks by Singular Value Decomposition (SVD) in multiple iterations. The RVQ's parametric solution assumes that data are subject to Gaussian distribution and uses Eigenvalue Allocation to get each stage-matrix {Rl}(1≤l≤L) at once, where L is the stage number of RVQ. Compared to various optimized PQ-based methods, our methods have good superiority on restoring quantization error.

Compact Representation of High-Dimensional Feature Vectors for Large-Scale Image Recognition and Retrieval.

In large-scale visual recognition and image retrieval tasks, feature vectors, such as Fisher vector (FV) or the vector of locally aggregated descriptors (VLAD), have achieved state-of-the-art results. However, the combination of the large numbers of examples and high-dimensional vectors necessitates dimensionality reduction, in order to reduce its storage and CPU costs to a reasonable range. In spite of the popularity of various feature compression methods, this paper shows that the feature (dimension) selection is a better choice for high-dimensional FV/VLAD than the feature (dimension) compression methods, e.g., product quantization. We show that strong correlation among the feature dimensions in the FV and the VLAD may not exist, which renders feature selection a natural choice. We also show that, many dimensions in FV/VLAD are noise. Throwing them away using feature selection is better than compressing them and useful dimensions altogether using feature compression methods. To choose features, we propose an efficient importance sorting algorithm considering both the supervised and unsupervised cases, for visual recognition and image retrieval, respectively. Combining with the 1-bit quantization, feature selection has achieved both higher accuracy and less computational cost than feature compression methods, such as product quantization, on the FV and the VLAD image representations.

Deep Quantization Network for Efficient Image Retrieval

Hashing has been widely applied to approximate nearest neighbor search for large-scale multimedia retrieval. Supervised hashing improves the quality of hash coding by exploiting the semantic similarity on data pairs and has received increasing attention recently. For most existing supervised hashing methods for image retrieval, an image is first represented as a vector of hand-crafted or machine-learned features, then quantized by a separate quantization step that generates binary codes. However, suboptimal hash coding may be produced, since the quantization error is not statistically minimized and the feature representation is not optimally compatible with the hash coding. In this paper, we propose a novel Deep Quantization Network (DQN) architecture for supervised hashing, which learns image representation for hash coding and formally control the quantization error. The DQN model constitutes four key components: (1) a sub-network with multiple convolution-pooling layers to capture deep image representations; (2) a fully connected bottleneck layer to generate dimension-reduced representation optimal for hash coding; (3) a pairwise cosine loss layer for similarity-preserving learning; and (4) a product quantization loss for controlling hashing quality and the quantizability of bottleneck representation. Extensive experiments on standard image retrieval datasets show the proposed DQN model yields substantial boosts over latest state-of-the-art hashing methods.

Multimodal medical image retrieval system

In this paper we depict an implemented system for medical image retrieval. Our system performs retrieval based on both textual and visual content, separately and combined, using advanced encoding and quantization techniques. The text-based retrieval subsystem uses textual data acquired from an image's corresponding article to generate a suitable representation. Using a vector space model, the generated representations structure is altered to increase performance. Query expansion with pseudo-relevance feedback is applied to fine-tune the results. The content-based retrieval subsystem performs retrieval based on visual features extracted from the images. A Gaussian Mixture Model is constructed from the extracted visual features, in our case - RGB histograms, and is used in encoding the same features into Fisher Vectors. With scalability and speed in mind, we utilized a product quantization technique over the generated vectors, which provides fast response times over large image collections. Product quantization drastically reduces the size of the image representation at almost no cost to accuracy, thus improving the scalability factor of our system. Our system uses modality classification to further improve retrieval results. This subsystem labels the image modality based on their visual content. The images are described using state-of-the-art opponentSIFT visual features. Classification was performed using Support Vector Machines (SVMs). The predictions from the SVMs are used for re-ranking the resulting images based on their modality and the modality of the query. The system was evaluated against the standardized ImageCLEF 2013, 2012 and 2011 medical datasets and it reported state-of-the-art performance for all datasets.

Segmentation-free Word Spotting for Handwritten Arabic Documents

In this paper we present an unsupervised segmentation-free method for spotting and searching query, especially, for images documents in handwritten Arabic, for this, Histograms of Oriented Gradients (HOGs) are used as the feature vectors to represent the query and documents image. Then, we compress the descriptors with the product quantization method. Finally, a better representation of the query is obtained by using the Support Vector Machines (SVM).

Circulant Temporal Encoding for Video Retrieval and Temporal Alignment

We address the problem of specific video event retrieval. Given a query video of a specific event, e.g., a concert of Madonna, the goal is to retrieve other videos of the same event that temporally overlap with the query. Our approach encodes the frame descriptors of a video to jointly represent their appearance and temporal order. It exploits the properties of circulant matrices to efficiently compare the videos in the frequency domain. This offers a significant gain in complexity and accurately localizes the matching parts of videos. The descriptors can be compressed in the frequency domain with a product quantizer adapted to complex numbers. In this case, video retrieval is performed without decompressing the descriptors. We also consider the temporal alignment of a set of videos. We exploit the matching confidence and an estimate of the temporal offset computed for all pairs of videos by our retrieval approach. Our robust algorithm aligns the videos on a global timeline by maximizing the set of temporally consistent matches. The global temporal alignment enables synchronous playback of the videos of a given scene.

Cache locality is not enough

Nearest Neighbor (NN) search in high dimension is an important feature in many applications (e.g., image retrieval, multimedia databases). Product Quantization (PQ) is a widely used solution which offers high performance, i.e., low response time while preserving a high accuracy. PQ represents high-dimensional vectors (e.g., image descriptors) by compact codes. Hence, very large databases can be stored in memory, allowing NN queries without resorting to slow I/O operations. PQ computes distances to neighbors using cache-resident lookup tables, thus its performance remains limited by (i) the many cache accesses that the algorithm requires, and (ii) its inability to leverage SIMD instructions available on modern CPUs. In this paper, we advocate that cache locality is not sufficient for efficiency. To address these limitations, we design a novel algorithm, PQ Fast Scan, that transforms the cache-resident lookup tables into small tables, sized to fit SIMD registers. This transformation allows (i) in-register lookups in place of cache accesses and (ii) an efficient SIMD implementation. PQ Fast Scan has the exact same accuracy as PQ, while having 4 to 6 times lower response time (e.g., for 25 million vectors, scan time is reduced from 74ms to 13ms).

Structure Sensitive Hashing With Adaptive Product Quantization.

Hashing has been proved as an attractive solution to approximate nearest neighbor search, owing to its theoretical guarantee and computational efficiency. Though most of prior hashing algorithms can achieve low memory and computation consumption by pursuing compact hash codes, however, they are still far beyond the capability of learning discriminative hash functions from the data with complex inherent structure among them. To address this issue, in this paper, we propose a structure sensitive hashing based on cluster prototypes, which explicitly exploits both global and local structures. An alternating optimization algorithm, respectively, minimizing the quantization loss and spectral embedding loss, is presented to simultaneously discover the cluster prototypes for each hash function, and optimally assign unique binary codes to them satisfying the affinity alignment between them. For hash codes of a desired length, an adaptive bit assignment is further appended to the product quantization of the subspaces, approximating the Hamming distances and meanwhile balancing the variance among hash functions. Experimental results on four large-scale benchmarks CIFAR-10, NUS-WIDE, SIFT1M, and GIST1M demonstrate that our approach significantly outperforms state-of-the-art hashing methods in terms of semantic and metric neighbor search.

Adaptive bit allocation product quantization

Product quantization (PQ) is a popular vector quantization method for approximate nearest neighbor search. The key idea of PQ is to decompose the original data space into the Cartesian product of some low-dimensional subspaces and then every subspace is quantized separately with the same number of codewords. However, the performance of PQ depends largely on the distribution of the original data. If the energies of subspaces are extremely unbalanced, PQ will achieve bad results. In this paper, we propose an adaptive bit allocation product quantization (BAPQ) method to deal with the problem of unbalanced energies of subspaces in PQ. In BAPQ, we adaptively allocate different numbers of codewords (or bits) to subspaces to quantize data for minimizing the total quantization distortion. The essence of our method is to find the optimal bit allocation scheme. To this end, we formulate an objective function about minimizing quantization distortion with respect to bit allocation scheme and adopt a greedy algorithm to find the near-optimal solution. BAPQ can achieve lower quantization distortion than PQ and optimized product quantization (OPQ). Besides, both bias and variance of difference between the true distance and the BAPQ׳s estimated distance are reduced from those of PQ and OPQ. Extensive experiments have verified the superiority of BAPQ over state-of-the-art approaches.

The Inverted Multi-Index.

A new data structure for efficient similarity search in very large datasets of high-dimensional vectors is introduced. This structure called the inverted multi-index generalizes the inverted index idea by replacing the standard quantization within inverted indices with product quantization. For very similar retrieval complexity and pre-processing time, inverted multi-indices achieve a much denser subdivision of the search space compared to inverted indices, while retaining their memory efficiency. Our experiments with large datasets of SIFT and GIST vectors demonstrate that because of the denser subdivision, inverted multi-indices are able to return much shorter candidate lists with higher recall. Augmented with a suitable reranking procedure, multi-indices were able to significantly improve the speed of approximate nearest neighbor search on the dataset of 1 billion SIFT vectors compared to the best previously published systems, while achieving better recall and incurring only few percent of memory overhead.

Sketch4Image: a novel framework for sketch-based image retrieval based on product quantization with coding residuals

Sketch-based Image Retrieval (SBIR) is one important branch of Content-based Image Retrieval (CBIR). SBIR means dealing with retrieval using simple edge or contour images. However, SBIR is more difficult than CBIR due to the lack of visual information, this makes the Bag-of-Words (BoW) or codebook in SBIR hard to construct. In this paper, we propose a novel SBIR framework based on Product Quantization (PQ) with sparse coding (SC) to construct an optimized codebook. By using state-of-the-art local descriptors, we transform sketch images into features and then build the optimized codebook using PQ-based SC. In the retrieval stage, we can obtain a better representation of the query sketch and testing images by the optimized codebook with coding quantization residuals, by which the information loss during feature encoding process can be reduced; similarity computing is implemented by comparing the feature histograms between a query sketch and the testing data for the final results. We demonstrate the superiority and effectiveness of the proposed SBIR by comparing it with several state-of-the-art methods on three public sketch datasets.

Multiproduct Uniform Polar Quantizer

The aim of this paper is to reduce the complexity of the unrestricted uniform polar quantizer (UUPQ), keeping its high performances. To achieve this, in this paper we propose the multiproduct uniform polar quan- tizer (MUPQ), where several consecutive magnitude levels are joined in segments and within each segment the uni- form product quantization is performed (i.e. all levels within one segments have the same number of phase levels). MUPQ is much simpler for realization than UUPQ, but it achieves similar performances as UUPQ. Since MUPQ has low complexity and achieves much better performances than the scalar uniform quantizer, it can be widely used instead of scalar uniform quantizers to im- prove performances, for any signal with the Gaussian distribution.