Product Quantization Research Articles

InMemQK: A Product Quantization Based MatMul Module for Compute-in-Memory Attention Macro

Large Language Models (LLMs), based on transformer architecture, have demonstrated remarkable capabilities in natural language processing tasks, enabling machines to generate human-like text and engage in meaningful dialogues. However, the exponential increase in model parameters has led to limitations in inference speed and energy efficiency. Compute-in-memory (CIM) technology offers a promising solution to accelerate AI inference by performing analog computations directly within memory, potentially reducing latency and power consumption. At the same time, CIM has been successfully applied to accelerate Convolutional Neural Networks (CNNs); however, the matrix–matrix multiplication (MatMul) operations inherent in the scaled dot-product attention of the transformer present unique challenges for direct CIM implementation. In this work, we propose InMemQK, a compute-in-memory-based attention accelerator that focuses on optimizing MatMul operations through software and hardware co-design. At the software level, InMemQK employs product quantization (PQ) to eliminate data dependencies. At the hardware level, InMemQK integrates energy-efficient time-domain MAC macros for ADC-free computations. Experimental results show InMemQK achieves 13.2×–13.9× lower power consumption than existing CIM-based accelerators.

Vector Databases

Vector databases represent a significant advancement in data management, tailored to efficiently handle high-dimensional vector embeddings. Unlike traditional databases, which face challenges with the complexity and scale of high-dimensional data, vector databases are engineered for optimal storage, indexing, and retrieval of vector data. This work explores the challenges posed by high-dimensional data, including the curse of dimensionality, and examines how vector databases address these issues through advanced indexing techniques such as Inverted File (IVF), Product Quantization (PQ), and Locality Sensitive Hashing (LSH). It highlights the importance of vector databases in facilitating rapid similarity searches, which are crucial for applications such as recommendation systems and search engines. The discussion also covers the evolution of vector databases and their impact on AI and machine learning.

Joint contrastive self-supervised learning and weak-orthogonal product quantization for fast image retrieval

Hashing and quantization methods based on deep learning are being widely used in large-scale image retrieval. Most methods use traditional supervised models to train models. Although these methods achieve excellent performance, they rely too much on expensive label information and do not fully utilize data resources. These methods overly encourage binary codes to retain original information, which may lead to models building too much useless information at the expense of discriminative information that is more important for retrieval. To address these issues, we propose a self-supervised method that does not require labeled information, called Contrastive Self-Supervised Weak-Orthogonal Product Quantization (CSWPQ). The method is trained in a label-free self-supervised manner to reduce the need for labels by maximizing the similarity of different views of the same image and minimizing the similarity of different image views, which not only improves the generalization ability of the model but also enhances the robustness. We introduce a quantified contrastive learning method that is different from traditional contrastive learning to learn more discriminative image representations by constructing quantified image features of different views of the image. In addition, in order to reduce the quantization error caused by product quantization, we impose a weak-orthogonal constraint to increase the diversity of representations, reduce the information loss and computational overhead in the quantization process, and improve the computational speed. Extensive experiments on CIFAR-10, NUS-WIDE and FLICKR25K datasets show that our proposed method achieves better performance.

A novel 3D LiDAR deep learning approach for uncrewed vehicle odometry.

Self-localization and pose registration are required for sound operation of next generation autonomous vehicles under uncertain environments. Thus, precise localization and mapping are crucial tasks in odometry, planning and other downstream processing. In order to reduce information loss in preprocessing, we propose leveraging LiDAR-based localization and mapping (LOAM) with point cloud-based deep learning instead of convolutional neural network (CNN) based methods that require cylindrical projection. The normal distribution transform (NDT) algorithm is then used to refine the former coarse pose estimation from the deep learning model. The results demonstrate that the proposed method is comparable in performance to recent benchmark studies. We also explore the possibility of using Product Quantization to improve NDT internal neighborhood searching by using high-level features as fingerprints.

A mobile edge computing-focused transferable sensitive data identification method based on product quantization

Sensitive data identification represents the initial and crucial step in safeguarding sensitive information. With the ongoing evolution of the industrial internet, including its interconnectivity across various sectors like the electric power industry, the potential for sensitive data to traverse different domains increases, thereby altering the composition of sensitive data. Consequently, traditional approaches reliant on sensitive vocabularies struggle to adequately address the challenges posed by identifying sensitive data in the era of information abundance. Drawing inspiration from advancements in natural language processing within the realm of deep learning, we propose a transferable Sensitive Data Identification method based on Product Quantization, named PQ-SDI. This innovative approach harnesses both the composition and contextual cues within textual data to accurately pinpoint sensitive information within the context of Mobile Edge Computing (MEC). Notably, PQ-SDI exhibits proficiency not only within a singular domain but also demonstrates adaptability to new domains following training on heterogeneous datasets. Moreover, the method autonomously identifies sensitive data throughout the entire process, eliminating the necessity for human upkeep of sensitive vocabularies. Extensive experimentation with the PQ-SDI model across four real-world datasets, resulting in performance improvements ranging from 2% to 5% over the baseline model and achieves an accuracy of up to 94.41%. In cross-domain trials, PQ-SDI achieved comparable accuracy to training and identification within the same domain. Furthermore, our experiments showcased the product quantization technique significantly reduces the parameter size by tens of times for the subsequent sensitive data identification phase, particularly beneficial for resource-constrained environments characteristic of MEC scenarios. This inherent advantage not only bolsters sensitive data protection but also mitigates the risk of data leakage during transmission, thus enhancing overall security measures in MEC environments.

HiHPQ: Hierarchical Hyperbolic Product Quantization for Unsupervised Image Retrieval

Existing unsupervised deep product quantization methods primarily aim for the increased similarity between different views of the identical image, whereas the delicate multi-level semantic similarities preserved between images are overlooked. Moreover, these methods predominantly focus on the Euclidean space for computational convenience, compromising their ability to map the multi-level semantic relationships between images effectively. To mitigate these shortcomings, we propose a novel unsupervised product quantization method dubbed Hierarchical Hyperbolic Product Quantization (HiHPQ), which learns quantized representations by incorporating hierarchical semantic similarity within hyperbolic geometry. Specifically, we propose a hyperbolic product quantizer, where the hyperbolic codebook attention mechanism and the quantized contrastive learning on the hyperbolic product manifold are introduced to expedite quantization. Furthermore, we propose a hierarchical semantics learning module, designed to enhance the distinction between similar and non-matching images for a query by utilizing the extracted hierarchical semantics as an additional training supervision. Experiments on benchmark image datasets show that our proposed method outperforms state-of-the-art baselines.

Expand-and-Quantize: Unsupervised Semantic Segmentation Using High-Dimensional Space and Product Quantization

Unsupervised semantic segmentation (USS) aims to discover and recognize meaningful categories without any labels. For a successful USS, two key abilities are required: 1) information compression and 2) clustering capability. Previous methods have relied on feature dimension reduction for information compression, however, this approach may hinder the process of clustering. In this paper, we propose a novel USS framework called Expand-and-Quantize Unsupervised Semantic Segmentation (EQUSS), which combines the benefits of high-dimensional spaces for better clustering and product quantization for effective information compression. Our extensive experiments demonstrate that EQUSS achieves state-of-the-art results on three standard benchmarks. In addition, we analyze the entropy of USS features, which is the first step towards understanding USS from the perspective of information theory.

Learnable product quantization for anomaly detection

In many anomaly detection applications, anomaly samples are difficult to obtain. We propose a novel product quantization (PQ)-based anomaly detection scheme: Learnable Product Quantization (LPQ), which only requires very few abnormal samples to train the model. The scheme extracts feature from high-dimensional data using the deep learning network, decomposes the feature space into a Cartesian product of low dimensional subspaces using PQ, and then produces sub-codebooks consisting of sub-codewords with clustering techniques. As a result, the extracted features with similar sub-vector are mapped into the same bucket, which reduces the time complexity of nearest neighbor retrieval significantly. In order to achieve reasonable codebooks, a PQ Table is embedded into the network. While training, we propose a novel metric learning strategy that makes the semantically similar (normal) samples closer and dissimilar (outlier) samples farther. The experimental results on benchmark datasets demonstrate that our metric learning strategy is better than the triplet loss and the sigmoid cross-entropy loss on the anomaly detection task. In general, LPQ shows excellent performance and high efficiency in anomaly detection.

Entropy-Optimized Deep Weighted Product Quantization for Image Retrieval.

Hashing and quantization have greatly succeeded by benefiting from deep learning for large-scale image retrieval. Recently, deep product quantization methods have attracted wide attention. However, representation capability of codewords needs to be further improved. Moreover, since the number of codewords in the codebook depends on experience, representation capability of codewords is usually imbalanced, which leads to redundancy or insufficiency of codewords and reduces retrieval performance. Therefore, in this paper, we propose a novel deep product quantization method, named Entropy Optimized deep Weighted Product Quantization (EOWPQ), which not only encodes samples into the weighted codewords in a new flexible manner but also balances the codeword assignment, improving while balancing representation capability of codewords. Specifically, we encode samples using the linear weighted sum of codewords instead of a single codeword as traditionally. Meanwhile, we establish the linear relationship between the weighted codewords and semantic labels, which effectively maintains semantic information of codewords. Moreover, in order to balance the codeword assignment, that is, avoiding some codewords representing most samples or some codewords representing very few samples, we maximize the entropy of the coding probability distribution and obtain the optimal coding probability distribution of samples by utilizing optimal transport theory, which achieves the optimal assignment of codewords and balances representation capability of codewords. The experimental results on three benchmark datasets show that EOWPQ can achieve better retrieval performance and also show the improvement of representation capability of codewords and the balance of codeword assignment.

Logit Variated Product Quantization Based on Parts Interaction and Metric Learning With Knowledge Distillation for Fine-Grained Image Retrieval

Logit Variated Product Quantization Based on Parts Interaction and Metric Learning With Knowledge Distillation for Fine-Grained Image Retrieval

Progressive Similarity Preservation Learning for Deep Scalable Product Quantization

Progressive Similarity Preservation Learning for Deep Scalable Product Quantization

Product Quantization for Limited Feedback MISO and RIS-Aided Wireless Communication Systems

Product Quantization for Limited Feedback MISO and RIS-Aided Wireless Communication Systems

Functional quantization of rough volatility and applications to volatility derivatives

We develop a product functional quantization of rough volatility. Since the optimal quantizers can be computed offline, this new technique, built on the insightful works by [Luschgy, H. and Pagès, G., Functional quantization of Gaussian processes. J. Funct. Anal., 2002, 196(2), 486–531; Luschgy, H. and Pagès, G., High-resolution product quantization for Gaussian processes under sup-norm distortion. Bernoulli, 2007, 13(3), 653–671; Pagès, G., Quadratic optimal functional quantization of stochastic processes and numerical applications. In Monte Carlo and Quasi-Monte Carlo Methods 2006, pp. 101–142, 2007 (Springer: Berlin Heidelberg)], becomes a strong competitor in the new arena of numerical tools for rough volatility. We concentrate our numerical analysis on the pricing of options on the VIX and realized variance in the rough Bergomi model [Bayer, C., Friz, P.K. and Gatheral, J., Pricing under rough volatility. Quant. Finance, 2016, 16(6), 887–904] and compare our results to other benchmarks recently suggested.

Optimization method of 3D reconstruction of metal cultural relics based on 3D laser scanning data reduction

AbstractThe extraction effect and feature matching are poor in the three‐dimensional reconstruction of metal cultural relics, which leads to a poor reconstruction effect. Therefore, an optimization method of three‐dimensional reconstruction of metal cultural relics based on 3D laser scanning data reduction is proposed. The overall technical design route of the method is shown as follows. Based on this model, the 3D laser scanning method is used to collect 3D images of metal artefacts, combined with colour space and 2D entropy detection methods to pre‐process 3D images, and feature matching of point clouds is carried out to extract and optimize the significant value of superpixels, and a 3D reconstructed visual model is constructed. Affine transformation is used to obtain the affine invariant moment of the structural light parameters of the visual feature lines of metal cultural relics. The light stability adjustment of the three‐dimensional reconstruction of metal cultural relics is realized by using the linear structural light adjustment method in the HSV colour space. The rigid and non‐rigid registration methods are introduced to match the point cloud. The product quantization algorithm is used to linearize the error function, and the block feature detection and matching model of spatial image is obtained. The noise number is judged by the threshold value, and the three‐dimensional reconstruction technology is combined to realize the three‐dimensional reconstruction of metal relics. The simulation results show that this method has good visual expression ability, high feature recognition rate, and improves the three‐dimensional reconstruction ability of metal relics.

Flexible product quantization for fast approximate nearest neighbor search

Flexible product quantization for fast approximate nearest neighbor search

Exploring Composite Indexes for Domain Adaptation in Neural Machine Translation

Domain adaptation in neural machine translation (NMT) tasks often involves working with datasets that have a different distribution from the training data. In such scenarios, k-nearest-neighbor machine translation (kNN-MT) has been shown to be effective in retrieving relevant information from large datastores. However, the high-dimensional context vectors of large neural machine translation model result in high computational costs for distance computation and storage. To address this issue, index optimization techniques have been proposed, including the use of inverted file index (IVF) and product vector quantization (PQ), called IVFPQ. In this paper, we explore the recent index techniques for efficient machine translation domain adaptation and combine multiple index structures to improve the efficiency of nearest-neighbor search in domain adaptation datasets for machine translation task. Specifically, we evaluate the effectiveness when combining optimized product quantization (OPQ) and hierarchical navigable small-world (HNSW) indexing with IVFPQ. Our study aims to provide insights into the most suitable composite index methods for efficient nearest-neighbor search in domain adaptation datasets, with a focus on improving both accuracy and speed.

Quantization to speedup approximate nearest neighbor search

The quantization-based approaches not only are the effective methods for solving the problems of approximate nearest neighbor search, but also effectively reduce storage space. However, many quantization-based approaches usually employ fixed nprobes to the search process for each query. This will lead to extra query consumption. Additionally, we observed that as the number of points in each cluster center of product quantization increases, the query cost also increases. To address this issue, we propose an acceleration strategy based on the IVF-HNSW framework to further speed up the query process. This strategy involves introducing an adaptive termination condition for queries and reducing the number of data points accessed by building HNSW results. Through extensive experiments, we have shown that our proposed method significantly accelerates the nearest neighbor search process.

Query-Aware Quantization for Maximum Inner Product Search

Maximum Inner Product Search (MIPS) plays an essential role in many applications ranging from information retrieval, recommender systems to natural language processing. However, exhaustive MIPS is often expensive and impractical when there are a large number of candidate items. The state-of-the-art quantization method of approximated MIPS is product quantization with a score-aware loss, developed by assuming that queries are uniformly distributed in the unit sphere. However, in real-world datasets, the above assumption about queries does not necessarily hold. To this end, we propose a quantization method based on the distribution of queries combined with sampled softmax. Further, we introduce a general framework encompassing the proposed method and multiple quantization methods, and we develop an effective optimization for the proposed general framework. The proposed method is evaluated on three real-world datasets. The experimental results show that it outperforms the state-of-the-art baselines.

SemDM: Task-oriented masking strategy for self-supervised visual learning

In this paper, we propose a novel learning scheme for better combining masked image modeling (MIM) and instance discrimination (ID). Motivated by compensating the requirement gap of masking strength between MIM and ID, we propose Semantic Disjoint Masking (SemDM), which decomposes the masking into two manners: preserving the majority of key patterns in images for ID, while dropping out most of them for MIM. Specifically, we utilize attention-guided masking in ID to help keeping the identity of object in image for encoder. While in MIM, we conversely only leave some hints about the object. Then these generated masked views only perform their specified learning task, facilitating more suitable visual priors to be learned in each learning task. Moreover, we introduce product quantization (PQ) to optimize the concept distributions in latent space, which guarantees that a compact set of meaningful visual concepts can be learned. Extensive experiments demonstrate that our method bootstraps meaningful visual concepts to guide visual understanding, and obtains state-of-the-art results on ImageNet-100.

Orthonormal product quantization network for scalable face image retrieval

Existing deep quantization methods provided an efficient solution for large-scale image retrieval. However, the significant intra-class variations, like pose, illumination, and expressions in face images, still pose a challenge. In light of this, face image retrieval requires sufficiently powerful learning metrics, which are absent in current deep quantization works. Moreover, to tackle the growing unseen identities in the query stage, face image retrieval drives more demands regarding model generalization and scalability than general image retrieval tasks. This paper integrates product quantization with orthonormal constraints into an end-to-end deep learning framework to effectively retrieve face images. Specifically, we propose a novel scheme that uses predefined orthonormal vectors as codewords to enhance the quantization informativeness and reduce codewords’ redundancy. A tailored loss function maximizes discriminability among identities in each quantization subspace for both the quantized and original features. An entropy-based regularization term is imposed to reduce the quantization error. Experiments are conducted on four commonly-used face datasets under both seen and unseen identity retrieval settings. Our method outperforms all the compared state-of-the-art under both settings. The proposed orthonormal codewords consistently boost both models’ standard retrieval performance and generalization ability, demonstrating the superiority of our method for scalable face image retrieval.