Inverted Index Research Articles

Optimizing Keyword Search Over Federated RDF Systems

The issue of keyword search in federated resource description framework(RDF) systems is revisited in this paper. Numerous independent SPARQL endpoints that solely offer SPARQL query interfaces make up a federated RDF system. Others find it challenging to remotely build up the global inverted indices over the full RDF network by downloading the dataset at various SPARQL endpoints. This poses a challenge for existing keyword search approaches. In this paper, we propose a keyword search approach for federated RDF systems without building up global inverted indices on the entire RDF graph. We build an offline schema graph for the federated RDF system. During query evaluation, the full-text search interfaces provided by existing SPARQL endpoints were used to map keywords to vertices in the schema graph. Subsequently, the schema graph was explored to construct SPARQL queries from the keywords. The constructed queries were evaluated over the underlying federated RDF systems. Some cost models were proposed to measure keyword mapping and query construction. On the other hand, to further improve the efficiency, a multiple query optimization strategy was designed to rewrite the queries and share the common computation during evaluation. Extensive experiments on real RDF datasets show that the proposed techniques are effective.

ENHANCING SEARCH CAPABILITIES: EXPLORING LUCENE AND SOLR TECHNIQUES FOR IMPROVED SEARCH PERFORMANCE

In todays digital age, data is everything. Dealing with and accessing a lot of data is a common task for almost every individual, be it for personal or for professional purposes and they expect applications to provide fast and accurate search capabilities to access and view their data. But all this data becomes meaningless if it cannot be accessed easily when needed or within a certain amount of time. This study aims to implement a powerful and customizable search feature in applications using the open-source search technologies, Lucene and Solr. The system will use Lucenes efficient indexing and querying capabilities to create an inverted index of the applications content, enabling fast and accurate full-text search. Solr will provide a web interface for the search functionality, allowing users to easily perform searches and filter results by various criteria such as date, author, category, and more. By incorporating Lucene and Solr, the system will be able to overcome the previous mentioned issues by providing search results significantly faster and deliver a robust, fast, and user-friendly search experience for applications that can be easily integrated into any application.

A novel cell partition method by introducing Silhouette Coefficient for fast approximate nearest neighbor search

In this paper, a novel cell partition method is proposed to accelerate the approximate nearest neighbor (ANN) search. The traditional cell-level elimination of database vectors, such as the inverted file system (IVF), has limitations on accelerating the search process. Therefore, the Silhouette Coefficient is introduced to realize more accurate and effective vector-level pruning. The value of the Silhouette Coefficient is used as a novel criterion for dividing the Voronoi cell into the center region and the border region. In addition, a dynamic threshold selection model is introduced to determine the sampling rate for each cell adaptively. The proposed method can greatly improve the search speed by reducing the vector-level distance calculation. Besides, the value of the Silhouette Coefficient is saved by offline generating a rank in each Voronoi cell, which does not bring any extra memory cost. Through experiments, it can be seen that for two widely-used datasets of SIFT1M and GIST1M, with no more than 1% accuracy reduction, the search speed can be significantly increased by about 44% compared with the standard inverted file system. And compared with state-of-the-art ANN search methods that aim at acceleration, the proposed method can also achieve higher search accuracy with almost the same computational consumption.

Efficient Secure Privacy Preserving Multi Keywords Rank Search over Encrypted Data in Cloud Computing

Verifiable multi keywords rank searchable encryption is an important technique to retrieve encrypted data in a cloud server and verify the validity of search results. The cloud server is malicious it may return incorrect search results or system fault. Thus, how to design verifiable secure multi keywords ranked search of outsourced encrypted data is an unsolved problem. Therefore, in this paper, we leverage the (T-SNE) techniques to reduce dimensionality and feathers transformation. Then adopt similar keywords clustering groups grouping strategy to construct an enhanced inverted index. Which links to coordinate matching techniques to improve accurate multi-keyword rank searches for practical user experience. Then Paillier Homorphic Encryption (P.H.E) and HMAC technique is adopted for search index encryption and keyword search verification to explore verifiable multi keywords rank search over encrypted data in the cloud server. The time cost of ranking score computation and search efficiency can be improved in two approaches, keywords grouping idea and no dummy keywords are added in query and documents vector. Thus, our scheme realizes verifiable multi keywords search exact ciphertext retrieval. We evaluated the performance of our proposed scheme using real-world datasets that ensure adequate security and privacy.

ColBERT-PRF: Semantic Pseudo-Relevance Feedback for Dense Passage and Document Retrieval

Pseudo-relevance feedback mechanisms, from Rocchio to the relevance models, have shown the usefulness of expanding and reweighting the users’ initial queries using information occurring in an initial set of retrieved documents, known as the pseudo-relevant set. Recently, dense retrieval – through the use of neural contextual language models such as BERT for analysing the documents’ and queries’ contents and computing their relevance scores – has shown a promising performance on several information retrieval tasks still relying on the traditional inverted index for identifying documents relevant to a query. Two different dense retrieval families have emerged: the use of single embedded representations for each passage and query, e.g., using BERT’s [CLS] token, or via multiple representations, e.g., using an embedding for each token of the query and document (exemplified by ColBERT). In this work, we conduct the first study into the potential for multiple representation dense retrieval to be enhanced using pseudo-relevance feedback and present our proposed approach ColBERT-PRF. In particular, based on the pseudo-relevant set of documents identified using a first-pass dense retrieval, ColBERT-PRF extracts the representative feedback embeddings from the document embeddings of the pseudo-relevant set. Among the representative feedback embeddings, the embeddings that most highly discriminate among documents are employed as the expansion embeddings, which are then added to the original query representation. We show that these additional expansion embeddings both enhance the effectiveness of a reranking of the initial query results as well as an additional dense retrieval operation. Indeed, experiments on the MSMARCO passage ranking dataset show that MAP can be improved by up to 26% on the TREC 2019 query set and 10% on the TREC 2020 query set by the application of our proposed ColBERT-PRF method on a ColBERT dense retrieval approach.We further validate the effectiveness of our proposed pseudo-relevance feedback technique for a dense retrieval model on MSMARCO document ranking and TREC Robust04 document ranking tasks. For instance, ColBERT-PRF exhibits up to 21% and 14% improvement in MAP over the ColBERT E2E model on the MSMARCO document ranking TREC 2019 and TREC 2020 query sets, respectively. Additionally, we study the effectiveness of variants of the ColBERT-PRF model with different weighting methods. Finally, we show that ColBERT-PRF can be made more efficient, attaining up to 4.54× speedup over the default ColBERT-PRF model, and with little impact on effectiveness, through the application of approximate scoring and different clustering methods.

DynamicRetriever: A Pre-trained Model-based IR System Without an Explicit Index

Web search provides a promising way for people to obtain information and has been extensively studied. With the surge of deep learning and large-scale pre-training techniques, various neural information retrieval models are proposed, and they have demonstrated the power for improving search (especially, the ranking) quality. All these existing search methods follow a common paradigm, i.e., index-retrieve-rerank, where they first build an index of all documents based on document terms (i.e., sparse inverted index) or representation vectors (i.e., dense vector index), then retrieve and rerank retrieved documents based on the similarity between the query and documents via ranking models. In this paper, we explore a new paradigm of information retrieval without an explicit index but only with a pre-trained model. Instead, all of the knowledge of the documents is encoded into model parameters, which can be regarded as a differentiable indexer and optimized in an end-to-end manner. Specifically, we propose a pre-trained model-based information retrieval (IR) system called DynamicRetriever, which directly returns document identifiers for a given query. Under such a framework, we implement two variants to explore how to train the model from scratch and how to combine the advantages of dense retrieval models. Compared with existing search methods, the model-based IR system parameterizes the traditional static index with a pre-training model, which converts the document semantic mapping into a dynamic and updatable process. Extensive experiments conducted on the public search benchmark Microsoft machine reading comprehension (MS MARCO) verify the effectiveness and potential of our proposed new paradigm for information retrieval.

Efficient immediate-access dynamic indexing

In a dynamic retrieval system, documents must be ingested as they arrive, and be immediately findable by queries. Our purpose in this paper is to describe an index structure and processing regime that accommodates that requirement for immediate access, seeking to make the ingestion process as streamlined as possible, while at the same time seeking to make the growing index as small as possible, and seeking to make term-based querying via the index as efficient as possible. We describe a new compression operation and a novel approach to extensible lists which together facilitate that triple goal. In particular, the structure we describe provides incremental document-level indexing using as little as two bytes per posting and only a small amount more for word-level indexing; provides fast document insertion; supports immediate and continuous queryability; provides support for fast conjunctive queries and similarity score-based ranked queries; and facilitates fast conversion of the dynamic index to a “normal” static compressed inverted index structure. Measurement of our new mechanism confirms that in-memory dynamic document-level indexes for collections into the gigabyte range can be constructed at a rate of two gigabytes/minute using a typical server architecture, that multi-term conjunctive Boolean queries can be resolved in just a few milliseconds each on average even while new documents are being concurrently ingested, and that the net memory space required for all of the required data structures amounts to an average of as little as two bytes per stored posting, less than half the space required by the best previous mechanism.

Public-Key Authenticated Encryption With Keyword Search Supporting Constant Trapdoor Generation and Fast Search

To improve the quality of medical care and reduce unnecessary medical errors, electronic medical records (EMRs) are widely applied in hospital information systems. However, rapidly increasing EMRs bring heavy storage burden to hospitals. Professional data management service provided by cloud server can save the hospital local storage, and meanwhile, realize EMRs sharing among external researchers. However, the risk of leaking information of patients discourages hospitals to outsource patients’ EMRs to the remote cloud server. In this paper, a secure and efficient cloud storing and sharing method can be achieved by applying the proposed public key authenticated encryption with ciphertext update and keyword search (PAUKS). The proposed PAUKS scheme enables EMRs to be encrypted and queried without decryption, and is secure against inside keyword guessing attacks. Compared with the recently proposed PAEKS in literature, the PAUKS scheme enjoys smaller computation and communication overheads. The required number of trapdoors per query is constant in PAUKS scheme, instead of the linearly expanding as the number of senders increases in PAEKS. Furthermore, an inverted index can be built safely in PAUKS scheme to accelerate the query procedure. Experiment results show that our PAUKS scheme owns a comparable running overhead, but enjoys a higher query efficiency after ciphertexts update.

Combining offline and on-the-fly disambiguation to perform semantic-aware XML querying

Many efforts have been deployed by the IR community to extend freetext query processing toward semi-structured XML search. Most methods rely on the concept of Lowest Comment Ancestor (LCA) between two or multiple structural nodes to identify the most specific XML elements containing query keywords posted by the user. Yet, few of the existing approaches consider XML semantics, and the methods that process semantics generally rely on computationally expensive word sense disambiguation (WSD) techniques, or apply semantic analysis in one stage only: performing query relaxation/refinement over the bag of words retrieval model, to reduce processing time. In this paper, we describe a new approach for XML keyword search aiming to solve the limitations mentioned above. Our solution first transforms the XML document collection (offline) and the keyword query (on-the-fly) into meaningful semantic representations using context-based and global disambiguation methods, specially designed to allow almost linear computation efficiency. We use a semantic-aware inverted index to allow semantic-aware search, result selection, and result ranking functionality. The semantically augmented XML data tree is processed for structural node clustering, based on semantic query concepts (i.e., key-concepts), in order to identify and rank candidate answer sub-trees containing related occurrences of query key-concepts. Dedicated weighting functions and various search algorithms have been developed for that purpose and will be presented here. Experimental results highlight the quality and potential of our approach.

Fast Algorithms for Denial Constraint Discovery

Denial constraints (DCs) are an integrity constraint formalism widely used to detect inconsistencies in data. Several algorithms have been devised to discover DCs from data, as manually specifying them is burdensome and, worse yet, error-prone. The existing algorithms follow two basic steps: building an intermediate data structure from records, then enumerating the DCs from that intermediate. However, current algorithms are often inefficient in computing these intermediates. Also, it is still unclear which enumeration algorithm performs best since some of the available algorithms have not yet been compared to each other. In response, we present a set of new algorithms with improved design choices. We introduce a parallel pipeline for rapidly computing the intermediate using custom data representations, algorithms, and indexes. For DC enumeration, we propose an inverted index, pruning, and parallel search strategies. We present hybrid approaches that integrate our techniques with previous enumeration algorithms, improving their performance in many scenarios. Our experimental study shows that the proposed DC discovery algorithms are consistently much faster (up to an order of magnitude) than the current state-of-the-art.

Aspect term extraction and optimized deep fuzzy clustering-based inverted indexing for document retrieval

Finding good relevant documents for query optimization is a well-known difficulty in the field of document retrieval. This paper develops a novel approach, named Exponential Aquila Optimizer (EAO)-based Deep Fuzzy Clustering for retrieving the documents. The proposed technique effectively finds the relevant documents and tries to understand the relationship among the documents and queries in terms of the significance of documents for query optimization. Here, the Deep Fuzzy Clustering is employed for performing cluster-based inverted indexing where the Training procedure of Deep Fuzzy Clustering is done using the developed optimization algorithm, named EAO. Meanwhile, the developed EAO is newly designed by the incorporation of EWMA and AO. In addition, complex query matching is done using the Tversky index for the user-based queries, such as multigram queries and semantic queries. On the other hand, the RV coefficient is accomplished for performing query optimization for relevant document retrieval. The proposed technique achieves better performance in terms of the performance metrics, like precision, recall, and F-measure with the maximum precision of 1, maximum recall of 0.956, and maximum F-measure of 0.977, respectively.

A new subsequence similarity retrieval method based on inverted index in EAST

The analysis technology of massive experimental data is one of the important concerns in the field of nuclear fusion research, and the waveform similarity retrieval is the basis of realizing data mining of discharge experiment. In this work, we proposed an efficient subsequence retrieval method for matching a selected waveform segment in the discharge signal waveform. This method accurately represents the waveform characteristics based on the slope of segment and the drop distance of segment head (DDSH). In order to improve the retrieval efficiency, an inverted index method based on the slope of segment was adopted to filter the discharge signal waveforms and then perform fine retrieving. The retrieval algorithm was implemented based on Python, and was verified with the plasma density and stored energy signals of historical experimental data from EAST mdsplus database. The test results show the retrieval efficiency is improved by 43.6% on average after using slope inverted index.

A Keyword-Grouping Inverted Index Based Multi-Keyword Ranked Search Scheme Over Encrypted Cloud Data

With the comprehensive development of cloud computing technology, more and more enterprises and individuals tend to outsource computing, data, and other resources to the cloud service providers to save the data management cost. Since the plaintext data outsourcing in the cloud could leak users’ private information, it is highly recommended to encrypt them before outsourcing. However, it is a challenge to perform searches over encrypted cloud data. In this paper, we adopt the keyword grouping idea into the traditional inverted index and propose a keyword-grouping inverted index (KGI-index). Based on the index, we propose a privacy-preserving KGI-index based multi-keywords ranked search scheme (KMRS). To improve the search efficiency, we adopt two strategies including grouping high relevant keywords and using the complete binary tree structure to optimize the index. The security analysis and experimental result show that the proposed scheme is a privacy-preserving and efficient multi-keyword ranked search scheme over encrypted cloud data.

Learning term weights by overfitting pairwise ranking loss

A search engine strikes a balance between effectiveness and efficiency to retrieve the best documents in a scalable way. Recent deep learning-based ranker methods are proving to be effective and improving the state-of-the-art in relevancy metrics. However, as opposed to index-based retrieval methods, neural rankers like bidirectional encoder representations from transformers (BERT) do not scale to large datasets. In this article, we propose a query term weighting method that can be used with a standard inverted index without modifying it. Query term weights are learned using relevant and irrelevant document pairs for each query, using a pairwise ranking loss. The learned weights prove to be more effective than term recall which is a probabilistic relevance feedback, previously used for the task. We further show that these weights can be predicted with a BERT regression model and improve the performance of both a BM25 based index and an index already optimized with a term weighting function.

A new fast inverted file-based algorithm for approximate nearest neighbor search without accuracy reduction

The inverted file (IVF) is one of the most efficient non-exhaustive search algorithms for approximate nearest neighbor (ANN) search. In this paper, we propose a new three-step fast search algorithm in the framework of IVF named fast IVF (fIVF) to accelerate the ANN search. Different from the traditional IVF algorithm that conducts a full search at the coarse quantization stage, we introduce (1) cell-level rejection, (2) vector-level rejection, and (3) subspace-based pyramid model with multi-resolution to speed up this high time-consuming stage. Since our proposed fIVF concentrates on finding the exact w nearest coarse codewords at the coarse quantization stage and follows the same product quantization stage, it can achieve the same search accuracy as the traditional IVF. Experimental results on different datasets demonstrate that our proposed fIVF can significantly speed up the search process while keeping the same search accuracy as the traditional IVF.

A Robust Coverless Steganography Scheme Using Camouflage Image

Recently, most coverless image steganography (CIS) methods are based on robust mapping rules. However, due to the limited mapping expression relationship between secret information and hash sequence, it is a challenge to further improve the hiding ability of coverless information hiding. Towards this goal, this paper proposes a robust coverless steganography scheme using camouflage image(CI-CIS). For the sender, CI-CIS introduces an camouflage image as the transmission carrier and establishes the correlation between them by Convolutional Neural Network(CNN) features. For the receiver, the camouflage image can retrive the corresponding stego-image to recover the secret information. To this end, we designed a reversible retrieval scheme between stego-image and camouflage image by using image clustering. At the same time, since the semantic features represented by CNN are robust to image attacks, our method can increase the capability of the CIS effectively. Besides, we also build an inverted index to improve retrieval efficiency. Experimental results and analysis show that the CI-CIS has higher robustness and more flexible capacity setting compared with the existing CIS methods.

Join optimization for inverted index technique on relational database management systems

In relational database management systems (RDBMSs), an efficient join method for text retrieval using an inverted index has been developed and implemented. However, the existing crossing of the posting inverted list increases the keyword search time for large texts because of unnecessary comparisons. The relation-based search produces results by utilizing the posting list intersection. To reduce the search time for queries, a multi-way skip-merge join algorithm is proposed in this study. The proposed algorithm improves the execution speed by using a sorted inverted index posting list to minimize unnecessary comparison operations in the posting list intersection. The skip-merge join method, which minimizes unnecessary comparison operations using the aggregate function, is integrated with the multi-way join as a replacement for the existing two-way join method. The join algorithm combining skip-merge join and multi-way join shows good performance because the number of search keywords and the number of documents increase. The performance improvement of the keyword search is verified by implementing the multi-way skip-merge join algorithm in PostgreSQL, an RDBMS.

Enabling Verifiable and Dynamic Ranked Search over Outsourced Data

Cloud computing as a promising computing paradigm is increasingly utilized as potential hosts for users&#x2019; massive dataset. Since the cloud service provider (CSP) is outside the users&#x2019; trusted domain, existing research suggests encrypting sensitive data before outsourcing and adopting Searchable Symmetric Encryption (SSE) to facilitate keyword-based searches over the ciphertexts. However, it remains a challenging task to design an effective SSE scheme that simultaneously supports <i>sublinear search time</i>, <i>efficient update and verification</i>, and <i>on-demand information retrieval</i>. To address this, we propose a Verifiable Dynamic Encryption with Ranked Search (VDERS) scheme that allows a user to perform top-<inline-formula><tex-math notation="LaTeX">$K$</tex-math><alternatives><mml:math><mml:mi>K</mml:mi></mml:math><inline-graphic xlink:href="liu-ieq1-2922177.gif"/></alternatives></inline-formula> searches on a <i>dynamic</i> document collection and verify the correctness of the search results in a secure and efficient way. Specifically, we first provide a basic construction, <inline-formula><tex-math notation="LaTeX">$\mathsf {VDERS}^0$</tex-math><alternatives><mml:math><mml:msup><mml:mi mathvariant="sans-serif">VDERS</mml:mi><mml:mn>0</mml:mn></mml:msup></mml:math><inline-graphic xlink:href="liu-ieq2-2922177.gif"/></alternatives></inline-formula>, where a <i>ranked inverted index</i> and a <i>verifiable matrix</i> are constructed to enable verifiable document insertion in top-<inline-formula><tex-math notation="LaTeX">$K$</tex-math><alternatives><mml:math><mml:mi>K</mml:mi></mml:math><inline-graphic xlink:href="liu-ieq3-2922177.gif"/></alternatives></inline-formula> searches. Then, an advanced construction, <inline-formula><tex-math notation="LaTeX">$\mathsf {VDERS}^{\star }$</tex-math><alternatives><mml:math><mml:msup><mml:mi mathvariant="sans-serif">VDERS</mml:mi><mml:mi>&#x2605;</mml:mi></mml:msup></mml:math><inline-graphic xlink:href="liu-ieq4-2922177.gif"/></alternatives></inline-formula>, is devised to further support document deletion with a reduced communication cost. Extensive experiments on real datasets demonstrate the efficiency and effectiveness of our VDERS scheme.

3iCubing: An Interval Inverted Index Approach to Data Cubes

The increase in the amounts of information used to analyze data is problematic since the memory necessary to store and process it is getting quite big. The interval inverted index representation was developed to reduce the required memory to store data, and Frag-Cubing is one of the most popular algorithms. In this paper, we propose two new data cubing algorithms: 3iCubing and M3iCubing. 3iCubing is a Frag-Cubing-based algorithm that uses the interval inverted index representation, while M3iCubing uses both a normal and interval inverted index data representation. The algorithms were compared using synthetic and real data sets in indexation and querying operations, both runtime and memory-wise. The experimental evaluation shows that 3iCubing can considerably reduce the memory needed to index a data set, reducing around 25% of the memory used by Frag-Cubing. Moreover, the results show that the interval inverted index representation is dependent on the data skewness to reduce the memory consumption, having positive results with highly skewed and real-world data sets.

Using Inverted Index for Fingerprint Search

Fingerprints are the most used biometric information for identifying people. With the increase in fingerprint data, indexing techniques are essential to perform an efficient search. In this work, we devise a solution that applies traditional inverted index, widely used in textual information retrieval, for fingerprint search. For that, it first converts fingerprints to text documents using techniques, such as Minutia Cylinder-Code and Locality-Sensitive Hashing, and then indexes them in inverted files. In the experimental evaluation, our approach obtained 0.42% of error rate with 10% of penetration rate in the FVC2002 DB1a data set, surpassing some established methods.