Inverted Index Research Articles

SPIRIT: Scalable and Persistent In-Memory Indices for Real-Time Search

Today, real-time search over big microblogging data requires low indexing and query latency. Online services, therefore, prefer to host inverted indices in memory. Unfortunately, as datasets grow, indices grow proportionally, and with limited DRAM scaling, the main memory faces high pressure. Also, indices must be persisted on disks as building them is computationally intensive. Consequently, it becomes necessary to frequently move on-heap index segments to storage, slowing down indexing. Reading storage-resident index segments requires filesystem calls and disk accesses during query evaluation, leading to high and unpredictable tail latency. This work exploits hybrid DRAM and scalable non-volatile memory (NVM) to offer dynamically growing and instantly searchable (i.e., real-time) persistent indices in on-heap memory. We implement SPIRIT, a real-time text inversion engine over hybrid memory. SPIRIT exploits the byte-addressability of hybrid memory to enable direct access to the index on a pre-allocated heap, eliminating expensive block storage accesses and filesystem calls during live operation. It uses an in-memory segment descriptor table to offer: ① instant segment availability to query evaluators upon fresh ingestion, ② low-overhead segment movement across memory tiers transparent to query evaluators, and ③ decoupled segment movement into NVM from their visibility to query evaluators, enabling different policies for mitigating NVM latency. SPIRIT accelerates compaction with zero-copy merging. It supports volatile, graceful shutdown, and crash-consistent indexing modes. The latter two modes offer instant recovery using persistent pointers. SPIRIT with hybrid memory and strong crash consistency guarantees exhibits many orders of magnitude better tail response times and query throughout than the state-of-the-art Lucene search engine. Compared against a highly optimized non-real-time evaluation of Lucene with liberal DRAM size, on average, across six query workloads, SPIRIT still delivers 2.5 × better (real-time) query throughput. Our work applies to other services that will benefit from direct on-heap access to large persistent indices.

Secure Blockchain Assisted Attribute-Based Keyword Search for Collaborative E-Healthcare

In the age of technological advancement, collaborative E-healthcare emerges as a transformative system eliminating traditional location and accessibility barriers in healthcare services. Here, Searchable Encryption (SE) plays a key role in enabling healthcare providers to outsource encrypted medical data and search services to third parties like cloud servers, thereby reducing storage and management expenses. This intermediary approach poses challenges of single-point failure, privacy breaches, and potentially untrustworthy results. State-of-the-art public key-based SE methods use a cloud-assisted architecture that doesn’t support reliable and practical searches with fine-grained permissions. Also, such systems require additional support to address potential privacy leakages and ensure data availability at the storage server. To address these concerns, we propose a B lockchain-assisted E fficient and S ecure K eyword S earch (BESKS) scheme to enforce fine-grained keyword search privilege control while achieving practical search complexity. Our scheme employs a ciphertext policy attribute-based keyword search mechanism where keywords are encrypted using expressive access policies to build an inverted index structure. The encrypted indexes are stored on the blockchain while encrypted medical documents are stored on InterPlanetary File System (IPFS) nodes to enhance availability and ensure the reliability and scalability of our approach. Our scheme utilizes blockchain-based smart contract for efficient, secure search operations and ensures financial fairness in fine-grained searches. Search tokens are generated based on user attributes and query keywords to facilitate private searches on-chain. To enhance the search process, our secure index enables exact match for a query keyword in constant time to ensure expensive authorization operations are performed only once. Theoretical analysis suggests that our BESKS is more efficient and secure than state-of-the-art schemes. Prototype implementation results on the Ethereum blockchain network further validate its feasibility for real-world applications, demonstrating the scheme's practical applicability in collaborative E-Healthcare systems.

Vector Databases

Vector databases represent a significant advancement in data management, tailored to efficiently handle high-dimensional vector embeddings. Unlike traditional databases, which face challenges with the complexity and scale of high-dimensional data, vector databases are engineered for optimal storage, indexing, and retrieval of vector data. This work explores the challenges posed by high-dimensional data, including the curse of dimensionality, and examines how vector databases address these issues through advanced indexing techniques such as Inverted File (IVF), Product Quantization (PQ), and Locality Sensitive Hashing (LSH). It highlights the importance of vector databases in facilitating rapid similarity searches, which are crucial for applications such as recommendation systems and search engines. The discussion also covers the evolution of vector databases and their impact on AI and machine learning.

Dear-PSM: A deep learning-based peptide search engine enables full database search for proteomics.

Peptide spectrum matching is the process of linking mass spectrometry data with peptide sequences. An experimental spectrum can match thousands of candidate peptides with variable modifications leading to an exponential increase in candidates. Completing the search within a limited time is a key challenge. Traditional searches expedite the process by restricting peptide mass errors and variable modifications, but this limits interpretive capability. To address this challenge, we propose Dear-PSM, a peptide search engine that supports full database searching. Dear-PSM does not restrict peptide mass errors, matching each spectrum to all peptides in the database and increasing the number of variable modifications per peptide from the conventional 3-20. Leveraging inverted index technology, Dear-PSM creates a high-performance index table of experimental spectra and utilizes deep learning algorithms for peptide validation. Through these techniques, Dear-PSM achieves a speed breakthrough 7 times faster than mainstream search engines on a regular desktop computer, with a remarkable 240-fold reduction in memory consumption. Benchmark test results demonstrate that Dear-PSM, in full database search mode, can reproduce over 90% of the results obtained by mainstream search engines when handling complex mass spectrometry data collected from different species using various instruments. Furthermore, it uncovers a substantial number of new peptides and proteins. Dear-PSM has been publicly released on the GitHub repository https://github.com/jianweishuai/Dear-PSM.

Extracting Key-phrase Embedding using Deep Average Network and Maximal Marginal Relevance to Enhance Information Retrieval

Background: Automatic keyphrase extraction (AKE) is essential to many NLP and information retrieval tasks. Extracting high-quality key phrases is difficult due to technological advancements and the exponential growth ‎of ‎textual data and digital sources. Unsupervised keyphrase extraction with cheap computing cost that relies ‎on ‎heuristic notions of phrase importance such as embedding similarities but their development necessitates in-depth subject expertise. Materials and Methods: This paper presents a method to obtain a semantic understanding of the query ‎and index documents by using ‎the embedding technique(universal Sentence ‎encoder (USE) ) while keeping the most informative ‎using Maximal Marginal Relevance (MMR) and then scoring(an ‎inverted index) the most documents relevant to the query ‎vector to improve the ‎performance of IR systems.‎ Results: The proposed retrieval model implement on the (Fire2011) dataset. The final ‎stage was evaluating the results of the baseline and the results (indexing and ranking) by using mean average precision (MAP). The ‎result of the baseline was 0.61, while the result ‎inverted index was 0.6277519 .‎ Conclusions: In this paper, we have discussed document retrieval using keep key phrases that ‎have ‏informativeness ‎properties by ‎using maximal ‎marginal ‎relevance, since if we extract a fixed number of top ‎keyphrases, ‎redundancy ‎hinders the diversification of the ‎extracted keyphrases.

Scalable Distributed Inverted List Indexes in Disaggregated Memory

Memory disaggregation separates compute (CPU) and main memory resources into disjoint physical units to enable elastic and independent scaling. Connected via high-speed RDMA-enabled networks, compute nodes can directly access remote memory. This setting often requires complex protocols with many network roundtrips as memory nodes have near-zero compute power. In this paper, we design a scalable distributed inverted list index for disaggregated memory architectures. An inverted list index maps a set of terms to lists of documents that contain this term. Current solutions either partition the index horizontally or vertically with severe limitations in the disaggregated memory setting due to data and access skew, high network latency, or out-of-memory errors. Our method partitions lists into fixed-size blocks and spreads them across the memory nodes to balance skewed accesses. Block-based list processing keeps the memory footprint of compute nodes low and masks latency by interleaving remote accesses with expensive list operations. In addition, we propose efficient updates with optimistic concurrency control and read-write conflict detection. Our experiments confirm the efficiency and scalability of our method.

From Text to Recommendations: How Vector Databases are Revolutionizing Personalized Content Delivery

Abstract: Effective similarity search and retrieval are now possible thanks to vector databases, which have become a potent tool for organizing and searching high-dimensional data. This article provides a thorough examination of vector databases, their underlying theories, and their applications across a range of industries. In handling complex data types, we address the significance of vector representation and emphasize the benefits of vector databases over traditional databases [1]. The article explores the process of creating a vector database, highlighting the critical function of indexing strategies such as IVF (Inverted File Indexing) and HNSW (Hierarchical Navigable Small World) in guaranteeing the effectiveness and precision of searches [2]. In addition, we discuss the problems caused by the curse of dimensionality and offer solutions to lessen its effects on nearest neighbor searches [3]

Fulgor: a fast and compact k-mer index for large-scale matching and color queries.

The problem of sequence identification or matching-determining the subset of reference sequences from a given collection that are likely to contain a short, queried nucleotide sequence-is relevant for many important tasks in Computational Biology, such as metagenomics and pangenome analysis. Due to the complex nature of such analyses and the large scale of the reference collections a resource-efficient solution to this problem is of utmost importance. This poses the threefold challenge of representing the reference collection with a data structure that is efficient to query, has light memory usage, and scales well to large collections. To solve this problem, we describe an efficient colored de Bruijn graph index, arising as the combination of a k-mer dictionary with a compressed inverted index. The proposed index takes full advantage of the fact that unitigs in the colored compacted de Bruijn graph are monochromatic (i.e., all k-mers in a unitig have the same set of references of origin, or color). Specifically, the unitigs are kept in the dictionary in color order, thereby allowing for the encoding of the map from k-mers to their colors in as little as 1 + o(1) bits per unitig. Hence, one color per unitig is stored in the index with almost no space/time overhead. By combining this property with simple but effective compression methods for integer lists, the index achieves very small space. We implement these methods in a tool called Fulgor, and conduct an extensive experimental analysis to demonstrate the improvement of our tool over previous solutions. For example, compared to Themisto-the strongest competitor in terms of index space vs. query time trade-off-Fulgor requires significantly less space (up to 43% less space for a collection of 150,000 Salmonella enterica genomes), is at least twice as fast for color queries, and is 2-6[Formula: see text] faster to construct.

Semi-supervised inverted file index approach for approximate nearest neighbor search

This paper introduces a novel modification to the Inverted File (IVF) index approach for approximate nearest neighbor search, incorporating supervised learning techniques to enhance the efficacy of intermediate clustering and achieve more balanced cluster sizes. The proposed method involves creating clusters using a neural network by solving a task to classify query vectors into the same bucket as their corresponding nearest neighbor vectors in the original dataset. When combined with minimizing the standard deviation of the bucket sizes, the indexing process becomes more efficient and accurate during the approximate nearest neighbor search. Through empirical evaluation on a test dataset, we demonstrate that the proposed semi-supervised IVF index approach outperforms the industry-standard IVF implementation with fixed parameters, including the total number of clusters and the number of clusters allocated to queries. This novel approach has promising implications for enhancing nearest-neighbor search efficiency in high-dimensional datasets across various applications, including information retrieval, natural language search, recommendation systems, etc.

Privacy-aware document retrieval with two-level inverted indexing

Previous work on privacy-aware ranking has addressed the minimization of information leakage when scoring top k documents, and has not studied on how to retrieve these top documents and their features for ranking. This paper proposes a privacy-aware document retrieval scheme with a two-level inverted index structure. In this scheme, posting records are grouped with bucket tags and runtime query processing produces query-specific tags in order to gather encoded features of matched documents with a privacy protection during index traversal. To thwart leakage-abuse attacks, our design minimizes the chance that a server processes unauthorized queries or identifies document sharing across posting lists through index inspection or across-query association. This paper presents the evaluation and analytic results of the proposed scheme to demonstrate the tradeoffs in its design considerations for privacy, efficiency, and relevance.

Improving Performance of Searchable Symmetric Encryption Through New Information Retrieval Scheme

Searchable symmetric encryption (SSE) is a robust cryptographic method that allows users to store and retrieve encrypted data on a remote server, such as a cloud server, while maintaining the privacy of the user's data. The technique employs symmetric encryption, which utilizes a single secret key for both data encryption and decryption. However, extensive research in this field has revealed that SSE encounters performance issues when dealing with large databases. Upon further investigation, it has become apparent that the issue is due to poor locality, necessitating that the cloud server access multiple memory locations for a single query. Additionally, prior endeavors in this domain centered on locality optimization have often led to expanded storage requirements (the stored encrypted index should not be substantially larger than the original index) or diminished data retrieval efficiency (only required data should be retrieved). We present a simple, secure, searchable, and cost-effective scheme, which addresses the aforementioned problems while achieving a significant improvement in information retrieval performance through site optimization by changing the encrypted inverted index storage mechanism. The proposed scheme has the optimal locality O(1) and the best read efficiency O(1) with no significant negative impact on the storage space, which often increases due to the improvement of the locality. Using real-world data, we demonstrate that our scheme is secure, practical, and highly accurate. Furthermore, our proposed work can resist well-known attacks such as keyword guessing attacks and frequency analysis attacks.

Fast and private multi-dimensional range search over encrypted data

For businesses looking to outsource their data to remote servers, cloud-based data storage is a popular choice. It is popular due to its flexibility, cost-effectiveness, and widespread availability. However, ensuring the confidentiality of data is a critical challenge that must be addressed. As a response to this issue, searchable encryption techniques have been developed. These techniques enable search queries to be performed on encrypted data while still keeping the plaintext confidential. While most existing symmetric searchable encryption schemes are designed for one-dimensional data records or document-keyword inverted indices, this paper introduces MDRSSE, a novel symmetric searchable encryption scheme specifically tailored for multi-dimensional range search. MDRSSE stands out as one of the pioneering SSE schemes to support multi-dimensional range search efficiently, without incurring undetermined additional communication or computation costs. By employing a single round of communication between the client and server, MDRSSE enables an honest-but-curious server to respond to multi-dimensional range queries without gaining knowledge of the data records or revealing the search query. Notably, MDRSSE boasts the lowest overall search complexity compared to existing state-of-the-art symmetric searchable encryption schemes designed for multi-dimensional range search. Extensive experimental tests were conducted to validate the robustness and practicality of our proposed scheme. The results demonstrate that, for a dataset consisting of 100K records with 12 dimensions (with each leaf node holding 500 records), it takes only 2.2 seconds to generate the encrypted dataset, and the overall setup phase completes within 2.5 seconds. Furthermore, for a range query encompassing 50 nodes, the search time is less than 2 ms and 3 ms for the client and server, respectively. MDRSSE achieves semantic security under the IND-CPA assumption, all without requiring additional storage size at the server.

String Matching Algorithm Using Multi-Characters Inverted Lists

This research article introduces a new string-matching algorithm that utilizes the multi-character inverted lists structure, implemented using a perfect hashing method. The new solution processes the pattern input string in a single pass. In the searching phase, the algorithm takes a linear time complexity. This approach demonstrates efficient and rapid performance in practical experiments when a binary character set is used. It also handles fewer characters and shorter lengths, even when the pattern length is increased. In some cases, this algorithm exhibits superior text search performance compared to other algorithms. It has been observed that this superiority is particularly evident when searching for smaller amounts of text. This algorithm can be explored more quickly, especially when the search factor is between 0.25 to 0.75 times the length of the pattern, approaching the performance of the traditional algorithm.

Dynamic forward secure searchable encryption scheme with phrase search for smart healthcare

The phrase searchable encryption scheme improves search efficiency and accuracy by searching a set of consecutive keywords from ciphertext. However, most current phrase searchable encryption schemes still don't support dynamic data updates in practical application scenarios such as smart healthcare due to forward security issues. In this article, a dynamic forward secure searchable encryption scheme with phrase search for smart healthcare systems is proposed. On the one hand, we adopt a state chain structure to construct an inverted index, which contains the location of keywords so as to achieve phrase search. On the other hand, we have added a modification operation that modifies the location of keywords directly by traversing the inverted index to achieve indexes update. In this way, the efficiency of data updating can be raised greatly because the update operation of first-delete-then-add is avoided in the previous phrase search scheme. The scheme is proved to satisfy the forward security in the leaked search pattern and access pattern since the server cannot know the updated state of updated files in the state chain index structure which is randomly generated by the client. Experimental results show that the proposed scheme has higher search accuracy and efficiency which can save at least 50 ms in search time compared to other related schemes.

Exploring Composite Indexes for Domain Adaptation in Neural Machine Translation

Domain adaptation in neural machine translation (NMT) tasks often involves working with datasets that have a different distribution from the training data. In such scenarios, k-nearest-neighbor machine translation (kNN-MT) has been shown to be effective in retrieving relevant information from large datastores. However, the high-dimensional context vectors of large neural machine translation model result in high computational costs for distance computation and storage. To address this issue, index optimization techniques have been proposed, including the use of inverted file index (IVF) and product vector quantization (PQ), called IVFPQ. In this paper, we explore the recent index techniques for efficient machine translation domain adaptation and combine multiple index structures to improve the efficiency of nearest-neighbor search in domain adaptation datasets for machine translation task. Specifically, we evaluate the effectiveness when combining optimized product quantization (OPQ) and hierarchical navigable small-world (HNSW) indexing with IVFPQ. Our study aims to provide insights into the most suitable composite index methods for efficient nearest-neighbor search in domain adaptation datasets, with a focus on improving both accuracy and speed.

HCV: Practical Multi-Keyword Conjunctive Query with Little Result Pattern Leakage

Multi-keyword conjunctive query is the most common searchable encryption (SE) scheme and gives practical search capability. This paper focuses on constructing a conjunctive query scheme with high privacy and accurate query result. For efficiency, this paper introduces a novel counter vector (CV) data structure instead of inverted index and builds the CV index database with three-tuple keywords. We use two kinds of cryptography primitives to encrypt the CV index database, respectively. One is symmetric encryption for efficiency, which gives two schemes, CVX and ICV. Experiments show that both CVX and ICV have much higher efficiency than the existing schemes. The other is BGN homomorphic encryption for privacy and gives the scheme HCV. HCV reduces much RP information leakage and even achieves “ideal” leakage for 3-keyword query. Experiments show that the HCV scheme achieves high privacy while compromising not so much storage.

Practical and Dynamic Attribute-Based Keyword Search Supporting Numeric Comparisons Over Encrypted Cloud Data

The attribute-based keyword search (ABKS), which simultaneously achieves searching and fine-grained access control over encrypted data, is frequently applied in cloud computing environments characterized by data storage and sharing. Recently, inspired by attribute-based encryption (ABE) and searchable encryption (SE) primitives, several ABKS schemes have been presented. However, almost all existing ABKS schemes actually only provide an attribute-based keyword equality match function and do not have a structural index to support practical search efficiency and dynamic data updates in real-world applications. To the best of our knowledge, this study is the first to realize an attribute-based keyword search construction supporting numerical comparison expressions with the practical search efficient and dynamic data update capacity (ABKS-NICEST), based on our proposed attribute-based keyword secure search scheme supporting numerical comparison expressions (ABKS-NICE) and an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">exclusive OR-chain</i> -based inverted index structure. To the best of our knowledge, ABKS-NICEST is the first attributed-based keyword search scheme with practical search efficiency and dynamic data update capacity. In addition, numerical values are an important and common attribute, so providing comparison expressions among numerical values can greatly enhance the expressivity of access policy. Therefore, we use the prefix membership verification technique to design a method to support any numeric comparison expression in a flexible and uniform manner. Through theoretical and experimental evaluations, we determine that ABKS-NICEST is the most efficient ABKS scheme.

Improving the Performance of Searchable Symmetric Encryption by Optimizing Locality

Both individuals and organizations prioritize data security and privacy, leading to an increasing focus on technological solutions that emphasize these aspects. Searchable symmetric encryption stands out as a prominent choice for secure storage and search of encrypted data on cloud servers, contributing to this objective. While searchable symmetric encryption offers numerous benefits, it also faces certain challenges, particularly when dealing with large databases. One significant challenge is poor performance, often attributed to poor locality. Visiting multiple locations can significantly increase the time required for data retrieval. Additionally, optimization methods aimed at improving locality can sometimes impact read efficiency or result in excessive storage of the encrypted index on the cloud server. In this paper, we present a scheme that effectively addresses these issues. We have enhanced the encrypted inverted index storage mechanism to improve information retrieval performance. Our scheme achieves optimal locality O(1) and optimal read efficiency in O(1), resulting in a significant increase in retrieval speed. Through experimentation with real-world data, we have demonstrated the practicality, accuracy, and security of our scheme.

An Attribute-Based Searchable Encryption Scheme for Cloud-Assisted IIoT

The searchable encryption is a particular case of structured encryption, which has been intensively researched in the secure cloud storage system. By constructing structured secure index such as encrypted multi-maps (EMM), encrypted inverted index (EII), etc., searchable encryption can achieve efficient keyword search over encrypted data set. However, existing searchable encryption constructions do not take search permissions into consideration, resulting in the lack of a mechanism of the data access control, which may not be suitable for IIoT applications, since an integrated industrial system contains all kinds of data with rigorous access permissions. In this paper, we construct an attribute-based searchable encryption construction for cloud-assisted IIoT application scenario. By designing the novel access policy based structured secure index and the attribute-based search token, our construction achieves fine-grained keyword search privilege control over encrypted IIoT data as well as the same search complexity as the traditional searchable encryption. To the best of our knowledge, this is the first Attribute-Based Searchable Encryption (ABSE) construction. We provide the correctness and security proofs for our construction. Experimental evaluation results in a real-world data set show the correctness and the practical search efficiency of the proposed ABSE.

ForestTI: A Scalable Inverted-Index-Oriented Timeseries Management System with Flexible Memory Efficiency

Timeseries management systems play an important role in IoT and performance monitoring. As the data volume scales up, absorbing data memory efficiently with high throughput becomes a growing requirement for timeseries management systems. However, the designs of the existing systems, especially the in-memory data structures, suffer from two issues. First, they suffer from the trade-off between memory efficiency and performance. Second, they are not scalable because of lock contention where they cannot benefit from parallel insertion and querying. In this paper, we propose ForestTI, a scalable inverted-index-oriented timeseries management system where the balance point between memory efficiency and performance can be flexibly adjusted under the increasing memory pressure. First, we present a two-level inverted index, which is scalable with optimistic lock coupling, and its internal structure can be gradually converted to more memory efficient representations. Second, we propose a two-level pointer swizzling mechanism to actively swap out the cold posting lists and in-memory timeseries objects as the number of timeseries increases. Finally, we further optimize the on-disk data structures (i.e. write-ahead logs and LSM-tree) to adapt to the high insertion throughput from the in-memory components. We prototype ForestTI with C++ from scratch, and compared to the storage engine of Prometheus, ForestTI achieves 1.79x higher insertion throughput, 52.1% lower query latency, and 56.9% lower memory occupation. We have released the open-source code of ForestTI for public access.