Inverted Index Research Articles

Evaluation strategies for top-k queries over memory-resident inverted indexes

Top-k retrieval over main-memory inverted indexes is at the core of many modern applications: from large scale web search and advertising platforms, to text extenders and content management systems. In these systems, queries are evaluated using two major families of algorithms: document-at-a-time (DAAT) and term-at-a-time (TAAT). DAAT and TAAT algorithms have been studied extensively in the research literature, but mostly in disk-based settings. In this paper, we present an analysis and comparison of several DAAT and TAAT algorithms used in Yahoo!'s production platform for online advertising. The low-latency requirements of online advertising systems mandate memory-resident indexes. We compare the performance of several query evaluation algorithms using two real-world ad selection datasets and query workloads. We show how some adaptations of the original algorithms for main memory setting have yielded significant performance improvement, reducing running time and cost of serving by 60% in some cases. In these results both the original and the adapted algorithms have been evaluated over memory-resident indexes, so the improvements are algorithmic and not due to the fact that the experiments used main memory indexes.

문헌정보학 학술지를 대상으로 한 온톨로지 구축에 관한 연구

This study constructed an ontology targeting journal articles and evaluated its performance. Also, the performance of a triple structure ontology was compared with the knowledge base of an inverted index file designed for a simple keyword search engine. The coverage was three years of articles published in the Journal of the Korean Society for Information Management from 2007 to 2009. Protege was used to construct an ontology, whilst utilizing an inverted index file to compare performance. The concept ontology was manually established, and the bibliography ontology was automatically constructed to produce an OWL concept ontology and an OWL bibliography ontology, respectively. This study compared the performance of the knowledge base of the ontology, using the Jena search engine with the performance of an inverted index file using the Lucene search engine. As a result, The Lucene showed higher precision rate, but Jena showed higher recall rate.

Textual Query of Personal Photos Facilitated by Large-Scale Web Data

The rapid popularization of digital cameras and mobile phone cameras has led to an explosive growth of personal photo collections by consumers. In this paper, we present a real-time textual query-based personal photo retrieval system by leveraging millions of Web images and their associated rich textual descriptions (captions, categories, etc.). After a user provides a textual query (e.g., “water”), our system exploits the inverted file to automatically find the positive Web images that are related to the textual query “water” as well as the negative Web images that are irrelevant to the textual query. Based on these automatically retrieved relevant and irrelevant Web images, we employ three simple but effective classification methods, k-Nearest Neighbor (kNN), decision stumps, and linear SVM, to rank personal photos. To further improve the photo retrieval performance, we propose two relevance feedback methods via cross-domain learning, which effectively utilize both the Web images and personal images. In particular, our proposed crossdomain learning methods can learn robust classifiers with only a very limited amount of labeled personal photos from the user by leveraging the prelearned linear SVM classifiers in real time. We further propose an incremental cross-domain learning method in order to significantly accelerate the relevance feedback process on large consumer photo databases. Extensive experiments on two consumer photo data sets demonstrate the effectiveness and efficiency of our system, which is also inherently not limited by any predefined lexicon.

An Effective Method of Arbitrary Length N-gram Statistics for Chinese Text

N-gram frequency is always an important indicator in corpus-based natural language processing. As the corpus being larger, time and space consumption is always a serious issue. In this paper, we proposed a two-stage inverted index based on bigrams. The proposed index is constructed by two levels of variable length vectors and the two characters of a bigram are hashed to different levels. In order to ensure an efficient indexing, some null units are reserved in the vectors. By this way, unigrams and bigrams can be indexed directly. With the help of the index, the frequency of arbitrary length ngram can be calculated efficiently by intersection operation. The two-stage index has excellent properties: (1) it can be constructed efficiently; (2) the space consumption is significantly reduced with no compression; (3) when n=1 and n=2, time complexity of statistical processing decreases significantly. When n>2, it is more efficient than suffix array in most cases. This method is suitable for Chinese and some other Asian languages.

Using Inverted Indices for Accelerating LINGO Calculations

The ever growing size of chemical databases calls for the development of novel methods for representing and comparing molecules. One such method called LINGO is based on fragmenting the SMILES string representation of molecules. Comparison of molecules can then be performed by calculating the Tanimoto coefficient, which is called LINGOsim when used on LINGO multisets. This paper introduces a verbose representation for storing LINGO multisets, which makes it possible to transform them into sparse fingerprints such that fingerprint data structures and algorithms can be used to accelerate queries. The previous best method for rapidly calculating the LINGOsim similarity matrix required specialized hardware to yield a significant speedup over existing methods. By representing LINGO multisets in the verbose representation and using inverted indices, it is possible to calculate LINGOsim similarity matrices roughly 2.6 times faster than existing methods without relying on specialized hardware.

Scalable, statistical storage allocation for extensible inverted file construction

An Inverted file is a commonly used index for both archival databases and free text where no updates are expected. Applications like information filtering and dynamic environments like the Internet require inverted files to be updated efficiently. Recently, extensible inverted files are proposed which can be used for fast online indexing. The effective storage allocation scheme for such inverted files uses the arrival rate to preallocate storage. In this article, this storage allocation scheme is improved by using information about both the arrival rates and their variability to predict the storage needed, as well as scaling the storage allocation by a logarithmic factor. The resultant, final storage utilization rate can be as high as 97–98% after indexing about 1.6 million documents. This compares favorably with the storage utilization rate of the original arrival rate storage allocation scheme. Our evaluation shows that the retrieval time for extensible inverted file on solid state disk is on average similar to the retrieval time for in-memory extensible inverted file. When file seek time is not an issue, our scalable storage allocation enables extensible inverted files to be used as the main index on disk. Our statistical storage allocation may be applicable to novel situations where the arrival of items follows a binomial, Poisson or normal distribution.

A Design of the Inverted Index Based on Web Document Comprehending

A design of inverted index based on web document comprehending is induced and the algorithm is presented by combining the inverted index and the web documents understanding technology. Experimental results show that less time is cost than the traditional inverted index to query in the documents with the same size.

Scalable Object Discovery: A Hash-Based Approach to Clustering Co-occurring Visual Words

We present a scalable approach to automatically discovering particular objects (as opposed to object categories) from a set of images. The basic idea is to search for local image features that consistently appear in the same images under the assumption that such co-occurring features underlie the same object. We first represent each image in the set as a set of visual words (vector quantized local image features) and construct an inverted file to memorize the set of images in which each visual word appears. Then, our object discovery method proceeds by searching the inverted file and extracting visual word sets whose elements tend to appear in the same images; such visual word sets are called co-occurring word sets. Because of unstable and polysemous visual words, a co-occurring word set typically represents only a part of an object. We observe that co-occurring word sets associated with the same object often share many visual words with one another. Hence, to obtain the object models, we further cluster highly overlapping co-occurring word sets in an agglomerative manner. Remarkably, we accelerate both extraction and clustering of co-occurring word sets by Min-Hashing. We show that the models generated by our method can effectively discriminate particular objects. We demonstrate our method on the Oxford buildings dataset. In a quantitative evaluation using a set of ground truth landmarks, our method achieved higher scores than the state-of-the-art methods.

2A1-L03 大規模地図検索のためのマルチスケールBag-Of-Features手法(ロボットビジョン)

Retrieving a large collection of environment maps built by mapper robots is a key problem for mobile robot self-localization. This map retrieval problem is addressed from a novel perspective of a multi-scale Bag-Of-Features (BOF) approach in the paper. In general, multi-scale approach is advantageous in capturing both the global structure and the local details of a given map. On the other hand, BOF map retrieval is advantageous in its compact map representation as well as efficient map retrieval using an inverted file system. Combining the advantages of both approaches is the main contribution of this paper. Our approach is based on multi-cue BOF as well as BOF dimension reduction, and achieves efficiency and compactness of the map retrieval system. Retrieval performance is evaluated using a large collection of point feature maps.

Product Quantization for Nearest Neighbor Search

This paper introduces a product quantization-based approach for approximate nearest neighbor search. The idea is to decompose the space into a Cartesian product of low-dimensional subspaces and to quantize each subspace separately. A vector is represented by a short code composed of its subspace quantization indices. The euclidean distance between two vectors can be efficiently estimated from their codes. An asymmetric version increases precision, as it computes the approximate distance between a vector and a code. Experimental results show that our approach searches for nearest neighbors efficiently, in particular in combination with an inverted file system. Results for SIFT and GIST image descriptors show excellent search accuracy, outperforming three state-of-the-art approaches. The scalability of our approach is validated on a data set of two billion vectors.

QUERI GANDA PADA SISTEM TEMU-KEMBALI INFORMASI BERBASIS JARINGAN INFERENSI

The MultipleQuery of Information Retrieval System Based on Inference Network. The multiple query is a query that combines Boolean and probabilistic query on the information retrieval inference network system, the system consists of two components i.e. a dokument and a query network, they are joined by links between the representation and query concepts. The document network build an inverted belief list and the query network are evaluated by using canonical matrix. The similarity process between representation and query conceps yields a set of relevant document retrieved. The experiment has showed that the use of multiple query formulations will significantly improve the retrieval perfomance, compared to either the Boolean or probabilistic query. Keywords: information retrieval, inverted list, link matrix, canonical matrix, precision-recall table.

Effectively Indexing the Uncertain Space

With the rapid development of various optical, infrared, and radar sensors and GPS techniques, there are a huge amount of multidimensional uncertain data collected and accumulated everyday. Recently, considerable research efforts have been made in the field of indexing, analyzing, and mining uncertain data. As shown in a recent book on uncertain data, in order to efficiently manage and mine uncertain data, effective indexing techniques are highly desirable. Based on the observation that the existing index structures for multidimensional data are sensitive to the size or shape of uncertain regions of uncertain objects and the queries, in this paper, we introduce a novel R-Tree-based inverted index structure, named UI-Tree, to efficiently support various queries including range queries, similarity joins, and their size estimation, as well as top-k range query, over multidimensional uncertain objects against continuous or discrete cases. Comprehensive experiments are conducted on both real data and synthetic data to demonstrate the efficiency of our techniques.

Predicate-based indexing for desktop search

Google and other products have revolutionized the way we search for information. There are, however, still a number of research challenges. One challenge that arises specifically in desktop search is to exploit the structure and semantics of documents, as defined by the application program that generated the data (e.g., Word, Excel, or Outlook). The current generation of search products does not understand these structures and therefore often returns wrong results. This paper shows how today’s search technology can be extended in order to take the specific semantics of certain structures into account. The key idea is to extend inverted file index structures with predicates which encode the circumstances under which certain keywords of a document become visible to a user. This paper provides a framework that allows to express the semantics of structures in documents and algorithms to construct enhanced, predicate-based indexes. Furthermore, this paper shows how keyword and phrase queries can be processed efficiently on such enhanced indexes. It is shown that the proposed approach has superior retrieval performance with regard to both recall and precision and has tolerable space and query running time overheads.

Using Multi-Modal Semantic Association Rules to fuse keywords and visual features automatically for Web image retrieval

A recent trend for image search is to fuse the two basic modalities of Web images, i.e., textual features (usually represented by keywords) and visual features for retrieval. The key issue is how to associate the two modalities for fusion. In this paper, a new approach based on Multi-Modal Semantic Association Rule (MMSAR) is proposed to fuse keywords and visual features automatically for Web image retrieval. A MMSAR contains a single keyword and several visual feature clusters, which crosses and associates the two modalities of Web images. A customized frequent itemsets mining algorithm is designed for the particular MMSARs based on the existing inverted file, and a new support–confidence framework is defined for the mining algorithm. Based on the mined MMSARs, the keywords and the visual features are fused automatically in the retrieval process. The proposed approach not only remarkably improves the retrieval precision, but also has fast response time. The experiments are carried out in a Web image retrieval system, VAST (VisuAl & SemanTic image search), and the results show the superiority and effectiveness of the proposed approach.

FragBag, an accurate representation of protein structure, retrieves structural neighbors from the entire PDB quickly and accurately

Fast identification of protein structures that are similar to a specified query structure in the entire Protein Data Bank (PDB) is fundamental in structure and function prediction. We present FragBag: An ultrafast and accurate method for comparing protein structures. We describe a protein structure by the collection of its overlapping short contiguous backbone segments, and discretize this set using a library of fragments. Then, we succinctly represent the protein as a "bags-of-fragments"-a vector that counts the number of occurrences of each fragment-and measure the similarity between two structures by the similarity between their vectors. Our representation has two additional benefits: (i) it can be used to construct an inverted index, for implementing a fast structural search engine of the entire PDB, and (ii) one can specify a structure as a collection of substructures, without combining them into a single structure; this is valuable for structure prediction, when there are reliable predictions only of parts of the protein. We use receiver operating characteristic curve analysis to quantify the success of FragBag in identifying neighbor candidate sets in a dataset of over 2,900 structures. The gold standard is the set of neighbors found by six state of the art structural aligners. Our best FragBag library finds more accurate candidate sets than the three other filter methods: The SGM, PRIDE, and a method by Zotenko et al. More interestingly, FragBag performs on a par with the computationally expensive, yet highly trusted structural aligners STRUCTAL and CE.

Exploiting geographic references of documents in a geographical information retrieval system using an ontology-based index

Both Geographic Information Systems and Information Retrieval have been very active research fields in the last decades. Lately, a new research field called Geographic Information Retrieval has appeared from the intersection of these two fields. The main goal of this field is to define index structures and techniques to efficiently store and retrieve documents using both the text and the geographic references contained within the text. We present in this paper two contributions to this research field. First, we propose a new index structure that combines an inverted index and a spatial index based on an ontology of geographic space. This structure improves the query capabilities of other proposals. Then, we describe the architecture of a system for geographic information retrieval that defines a workflow for the extraction of the geographic references in documents. The architecture also uses the index structure that we propose to solve pure spatial and textual queries as well as hybrid queries that combine both a textual and a spatial component. Furthermore, query expansion can be performed on geographic references because the index structure is based in an ontology.

관계형 DBMS 기반의 XML 데이터를 위한 k-비트맵 클러스터링 기법

웹2.0 환경의 발달과 함께 XML 데이터의 사용도 증가하였는데, 특히 블로그나 뉴스 피드의 정보 전달을 위한 RSS나 ATOM 포맷의 기반 기술로 사용되면서 그 장점과 가치를 인정받고 있다. XML 데이터의 인덱싱을 위한 여러 기법들 중 빠른 검색성능을 보인 비트맵 클러스터링은 관계형 DBMS를 기반으로 메모리에 인덱스를 유지하는 기법이다. 기존의 비트맵 클러스터링 기법을 이용하여 XML 데이터를 인덱싱할 때 너무 많은 클러스터가 생성되어 오히려 검색 효율과 결과의 품질이 저하되는 문제점이 있었다. 본 논문에서는 이 문제점을 해결하기 위해 사용자가 제시하는 k개의 클러스터를 생성하는 k-비트맵 클러스터링 기법과 대표비트를 생성할 때 배제된 단어를 검색하기 위한 역인덱스를 함께 유지하는 방법을 제안한다. 성능평가를 수행한 결과 제안하는 기법은 생성되는 클러스터의 수를 임의로 설정할 수 있을 뿐만 아니라 단일 단어 검색에서 높은 재현율을 보였고, 2개의 인덱스를 함께 유지할 때에는 질의에 대해 모든 관련된 문서의 반환을 보장하였다. Use of XML data has been increased with growth of Web 2.0 environment. XML is recognized its advantages by using based technology of RSS or ATOM for transferring information from blogs and news feed. Bitmap clustering is a method to keep index in main memory based on Relational DBMS, and which performed better than the other XML indexing methods during the evaluation. Existing method generates too many clusters, and it causes deterioration of result of searching quality. This paper proposes k-Bitmap clustering method that can generate user defined k clusters to solve above-mentioned problem. The proposed method also keeps additional inverted index for searching excluded terms from representative bits of k-Bitmap. We performed evaluation and the result shows that the users can control the number of clusters. Also our method has high recall value in single term search, and it guarantees the searching result includes all related documents for its query with keeping two indices.

Introduction to Search 2009

Search is like the blind men and the elephant. The story goes that four blind men approached an elephant. By touching the hide, trunk, tail and legs each thought of the elephant as an entirely different animal. From the hide we can conclude that search is a contextual process where searchers must aimlessly feel around until they either accidentally find what they want or give up. From the trunk we can conclude that search should be like a fire hose where we flood the searchers with content, overwhelming them. The tail? From this perspective search is seen as the infamous “long tail” wherein searchers are fed minuscule tidbits of esoterica, leaving them curious, but famished.And those feeling the legs would conclude that we should stomp the content into submission, pulverizing until it is unrecognizable.

Improving Bag-of-Features for Large Scale Image Search

This article improves recent methods for large scale image search. We first analyze the bag-of-features approach in the framework of approximate nearest neighbor search. This leads us to derive a more precise representation based on Hamming embedding (HE) and weak geometric consistency constraints (WGC). HE provides binary signatures that refine the matching based on visual words. WGC filters matching descriptors that are not consistent in terms of angle and scale. HE and WGC are integrated within an inverted file and are efficiently exploited for all images in the dataset. We then introduce a graph-structured quantizer which significantly speeds up the assignment of the descriptors to visual words. A comparison with the state of the art shows the interest of our approach when high accuracy is needed. Experiments performed on three reference datasets and a dataset of one million of images show a significant improvement due to the binary signature and the weak geometric consistency constraints, as well as their efficiency. Estimation of the full geometric transformation, i.e., a re-ranking step on a short-list of images, is shown to be complementary to our weak geometric consistency constraints. Our approach is shown to outperform the state-of-the-art on the three datasets.

Indexing Boolean expressions

We consider the problem of efficiently indexing Disjunctive Normal Form (DNF) and Conjunctive Normal Form (CNF) Boolean expressions over a high-dimensional multi-valued attribute space. The goal is to rapidly find the set of Boolean expressions that evaluate to true for a given assignment of values to attributes. A solution to this problem has applications in online advertising (where a Boolean expression represents an advertiser's user targeting requirements, and an assignment of values to attributes represents the characteristics of a user visiting an online page) and in general any publish/subscribe system (where a Boolean expression represents a subscription, and an assignment of values to attributes represents an event). All existing solutions that we are aware of can only index a specialized sub-set of conjunctive and/or disjunctive expressions, and cannot efficiently handle general DNF and CNF expressions (including NOTs) over multi-valued attributes. In this paper, we present a novel solution based on the inverted list data structure that enables us to index arbitrarily complex DNF and CNF Boolean expressions over multi-valued attributes. An interesting aspect of our solution is that, by virtue of leveraging inverted lists traditionally used for ranked information retrieval, we can efficiently return the top-N matching Boolean expressions. This capability enables emerging applications such as ranked publish/subscribe systems [16], where only the top subscriptions that match an event are desired. For example, in online advertising there is a limit on the number of advertisements that can be shown on a given page and only the "best" advertisements can be displayed. We have evaluated our proposed technique based on data from an online advertising application, and the results show a dramatic performance improvement over prior techniques.