Inverted Index Research Articles

Exploring Linked Data with Contextual Tag Clouds

In this paper we present the contextual tag cloud system: a novel application that helps users explore a large scale RDF dataset. Unlike folksonomy tags used in most traditional tag clouds, the tags in our system are ontological terms (classes and properties), and a user can construct a context with a set of tags that defines a subset of instances. Then in the contextual tag cloud, the font size of each tag depends on the number of instances that are associated with that tag and all tags in the context. Each contextual tag cloud serves as a summary of the distribution of relevant data, and by changing the context, the user can quickly gain an understanding of patterns in the data. Furthermore, the user can choose to include RDFS taxonomic and/or domain/range entailment in the calculations of tag sizes, thereby understanding the impact of semantics on the data. In this paper, we describe how the system can be used as a query building assistant, a data explorer for casual users, or a diagnosis tool for data providers. To resolve the key challenge of how to scale to Linked Data, we combine a scalable pre-processing approach with a specially-constructed inverted index, use three approaches to prune unnecessary counts for faster online computations, and design a paging and streaming interface. Together, these techniques enable a responsive system that in particular holds a dataset with more than 1.4 billion triples and over 380,000 tags. Via experimentation, we show how much our design choices benefit the responsiveness of our system.

Exploring Linked Data with contextual tag clouds

In this paper we present the contextual tag cloud system: a novel application that helps users explore a large scale RDF dataset. Unlike folksonomy tags used in most traditional tag clouds, the tags in our system are ontological terms (classes and properties), and a user can construct a context with a set of tags that defines a subset of instances. Then in the contextual tag cloud, the font size of each tag depends on the number of instances that are associated with that tag and all tags in the context. Each contextual tag cloud serves as a summary of the distribution of relevant data, and by changing the context, the user can quickly gain an understanding of patterns in the data. Furthermore, the user can choose to include RDFS taxonomic and/or domain/range entailment in the calculations of tag sizes, thereby understanding the impact of semantics on the data. In this paper, we describe how the system can be used as a query building assistant, a data explorer for casual users, or a diagnosis tool for data providers. To resolve the key challenge of how to scale to Linked Data, we combine a scalable preprocessing approach with a specially-constructed inverted index, use three approaches to prune unnecessary counts for faster online computations, and design a paging and streaming interface. Together, these techniques enable a responsive system that in particular holds a dataset with more than 1.4 billion triples and over 380,000 tags. Via experimentation, we show how much our design choices benefit the responsiveness of our system.

EFFICIENTLY SEARCHING NEAREST NEIGHBOR IN DOCUMENTS USING KEYWORDS

Conservative spatial queries, such as range search and nearest neighbor reclamation, involve only cond itions on objects’ numerical properties. Today, many modern applications call fo r novel forms of queries that aim to find objects s atisfying both a spatial predicate, and a predicate on their associated text s. For example, instead of considering all the rest aurants, a nearest neighbor query would instead ask for the restaurant that is the cl osest among those whose menus contain “steak, spagh etti, brandy” all at the same time. Currently the best solution to such queries ibased on the InformationRetrieval2-tree, which, h as a few deficiencies that seriously impact its efficiency. Motivated by this, there is a development of a new access method call ed the spatial inverted index that extends the conventional inverted index to cope wit h multidimensional data, and comes with algorithms that can answer nearest neighbor queries with keywords in real time. As ver ified by experiments, the proposed techniques outpe rform the InformationRetrieval2-tree in query response time soften by a factor of orders of magnit ude

Self-similarity-based partial near-duplicate video retrieval and alignment

There have been recent studies on partial near-duplicate videos, which involve segments of videos that are near duplicates of each other. State-of-the-art searching schemes usually segment the input video into clips and implement clip-level near-duplicate retrieval. However, the segmentation results are always poorly aligned, which lead to a difficult “unbalance” problem. In this paper, we introduce a self-similarity-based feature representation called the Self-Similarity Belt (SSBelt), which derives from the Self-Similarity Matrix (SSM). In addition, a distinctive pattern in SSBelt called the Interest Corner is detected and described by a bag-of-words representation. The visual words are then combined into visual shingles and indexed by an inverted file index for fast retrieval. Another important task is to accurately align the unbalanced clips, for which we propose the Intensity Mark (IMark) and design a coarse-to-fine near-duplicate video localization scheme. Experimental results show the effectiveness of our approach for both web-based near-duplicate video and unbalanced video datasets. The near-duplicate alignment capacity of IMark is also shown to be effective.

GPS Estimation for Places of Interest From Social Users' Uploaded Photos

Social media has become a very popular way for people to share their photos with friends. Because most of the social images are attached with GPS (geo-tags), a photo's GPS information can be estimated with the help of the large geo-tagged image set while using a visual searching based approach. This paper proposes an unsupervised image GPS location estimation approach with hierarchical global feature clustering and local feature refinement. It consists of two parts: an offline system and an online system. In the offline system, a hierarchical structure is constructed for a large-scale offline social image set with GPS information. Representative images are selected for each GPS location refined cluster, and an inverted file structure is proposed. In the online system, when given an input image, its GPS information can be estimated by hierarchical global clusters selection and local feature refinement in the online system. Both the computational cost and GPS estimation performance demonstrates the effectiveness of the proposed hierarchical structure and inverted file structure in our approach.

Scanning Compressed Full Text Files

From the 1994 CAIS Conference: The Information Industry in Transition McGill University, Montreal, Quebec. May 25 - 27, 1994.In this paper we discuss an application of compression, not with the overall goal of reducing disk space, but with the goal of extending the applicability of full text scan procedures to larger text files for use in on-line search environments. This paper presents an alternative to inverted file generation for access to full text data files of medium size, 50-200Mbytes, for which the cost of generating a full inverted file is not warranted. Full scan techniques, which are often useful in interactive situations for small files, become unacceptably slow for interactive sessions with files above 10 Mbytes and so the use of compression to reduce the quantity of data scanned is an attractive alternative. Furthermore, an index that can be used to reduce search time further to very acceptable 1-2 second times can be generated as a byproduct of the compression.

Data Mining Model for the Data Retrieval from Central Server Configuration

A server, which is to keep track of heavy document traffic, is unable to filter the documents that are most relevant and updated for continuous text search queries. This paper focuses on handling continuous text extraction sustaining high document traffic. The main objective is to retrieve recent updated documents that are most relevant to the query by applying sliding window technique. Our solution indexes the streamed documents in the main memory with structure based on the principles of inverted file, and processes document arrival and expiration events with incremental threshold-based method. It also ensures elimination of duplicate document retrieval using unsupervised duplicate detection. The documents are ranked based on user feedback and given higher priority for retrieval.

New inverted index storage scheme for Chinese search engine

After analyzing inverted index structure and access mode of an open source search engine-ASPSeek,this paper gave an Abstract definition of "inverted index".In order to solve the difficulties of inverted index updating and the efficiency issues caused by directly accessing inverted index through file caching of operating system in ASPSeek,considering the characteristics of 1.25 million Chinese agricultural Web pages,this article proposed a new blocking inverted index storage scheme with a buffer mechanism which was based on CLOCK replacement algorithm.The experimental results show that the new scheme is more efficient than ASPSeek whether the buffer system is disabled or enabled.When the buffer system got enabled and 160 thousand Chinese terms or 50 thousand high-frequency Chinese terms were used as a test set,the retrieval time of new scheme tended to be a constant after one million accesses.Even when using entire 827 309 terms as a test set,the retrieval time of new scheme began to converge after two million accesses.

Improving the performance of pipelined query processing with skipping—and its comparison to document-wise partitioning

Web search engines need to provide high throughput and short query latency. Recent results show that pipelined query processing over a term-wise partitioned inverted index may have superior throughput. However, the query processing latency and scalability with respect to the collections size are the main challenges associated with this method. In this paper, we evaluate the effect of inverted index skipping on the performance of pipelined query processing. Further, we introduce a novel idea of using Max-Score pruning within pipelined query processing and a new term assignment heuristic, partitioning by Max-Score. Our current results indicate a significant improvement over the state-of-the-art approach and lead to several further optimizations which include dynamic load balancing, intra-query concurrent processing and a hybrid combination between pipelined and non-pipelined execution. Lastly, we show how the state of term-wise partitioning relates to the industry standard document-wise partitioning. Even though there are situations pipelined query processing is advantegous, document-wise partitioning is still the road to follow.

Effective ranking and search techniques for Web resources considering semantic relationships

On the Semantic Web, the types of resources and the semantic relationships between resources are defined in an ontology. By using that information, the accuracy of information retrieval can be improved.In this paper, we present effective ranking and search techniques considering the semantic relationships in an ontology. Our technique retrieves top-k resources which are the most relevant to query keywords through the semantic relationships. To do this, we propose a weighting measure for the semantic relationship. Based on this measure, we propose a novel ranking method which considers the number of meaningful semantic relationships between a resource and keywords as well as the coverage and discriminating power of keywords. In order to improve the efficiency of the search, we prune the unnecessary search space using the length and weight thresholds of the semantic relationship path. In addition, we exploit Threshold Algorithm based on an extended inverted index to answer top-k results efficiently. The experimental results using real data sets demonstrate that our retrieval method using the semantic information generates accurate results efficiently compared to the traditional methods.

A term-based inverted index partitioning model for efficient distributed query processing

In a shared-nothing, distributed text retrieval system, queries are processed over an inverted index that is partitioned among a number of index servers. In practice, the index is either document-based or term-based partitioned. This choice is made depending on the properties of the underlying hardware infrastructure, query traffic distribution, and some performance and availability constraints. In query processing on retrieval systems that adopt a term-based index partitioning strategy, the high communication overhead due to the transfer of large amounts of data from the index servers forms a major performance bottleneck, deteriorating the scalability of the entire distributed retrieval system. In this work, to alleviate this problem, we propose a novel inverted index partitioning model that relies on hypergraph partitioning. In the proposed model, concurrently accessed index entries are assigned to the same index servers, based on the inverted index access patterns extracted from the past query logs. The model aims to minimize the communication overhead that will be incurred by future queries while maintaining the computational load balance among the index servers. We evaluate the performance of the proposed model through extensive experiments using a real-life text collection and a search query sample. Our results show that considerable performance gains can be achieved relative to the term-based index partitioning strategies previously proposed in literature. In most cases, however, the performance remains inferior to that attained by document-based partitioning.

Research on Efficient Spatial Keyword Queries Supporting Wildcard

With the popularity of location-based services, Web contents are being geo-tagged and spatial keyword queries that retrieve objects satisfying both spatial and keyword conditions are gaining in prevalence. The existing spatial keyword queries focus on exact match or prefix match of the keywords cannot satisfy the demand of wildcard based imprecise match in many realistic scenes. Aiming to solve this problem, two methods which are fit for different situation are put forward: the inverted file and R-tree integrated index which fits for the situation that requires high time efficiency and the Prefix Bloom Filter and R-tree integrated index which fits for the situation requiring high space efficiency. The effectiveness of the two indexes is valid through experiments.

A Mathematical Formula Retrieval Method Using Structure Sub-Tree

It is quite inadequate in providing formula retrieval function by traditional retrieval techniques used in full-text information retrieval system. The main reason is that there are many difficulties to extract the keywords of the mathematical formulas. In this paper, a detailed analysis of the structural characteristics of mathematical formulas and existing index mechanism of mathematical formula searching engine is fulfilled. Then a full-text index (named SLIndex) of mathematical formulas with B+ tree structure is designed and implemented which extracts the structured logic sub-tree feature as keywords of formulas and employs inverted index. Finally, a formula search engine model based on SLIndex is implemented in Apache 2.0 web server.

ObjectPatchNet: Towards scalable and semantic image annotation and retrieval

The ever increasing Internet image collection densely samples the real world objects, scenes, etc. and is commonly accompanied with multiple metadata such as textual descriptions and user comments. Such image data has potential to serve as a knowledge source for large-scale image applications. Facilitated by such publically available and ever-increasing loosely annotated image data on the Internet, we propose a scalable data-driven solution for annotating and retrieving Web-scale image data. We extrapolate from large-scale loosely annotated images a compact and informative representation, namely ObjectPatchNet. Each vertex in ObjectPatchNet, which is called as an ObjectPatchNode, is defined as a collection of discriminative image patches annotated with object category labels. The edge linking two ObjectPatchNodes models the co-occurrence relationship among different objects in the same image. Therefore, ObjectPatchNet models not only probabilistically labeled image patches, but also the contextual relationship between objects. It is well suited to scalable image annotation task. Besides, we further take ObjectPatchNet as a visual vocabulary with semantic labels, and hence are able to easily develop inverted file indexing for efficient semantic image retrieval. ObjectPatchNet is tested on both large-scale image annotation and large-scale image retrieval applications. Experimental results manifest that ObjectPatchNet is both discriminative and efficient in these applications.

Scalable Face Image Retrieval Using Attribute-Enhanced Sparse Codewords

Photos with people (e.g., family, friends, celebrities, etc.) are the major interest of users. Thus, with the exponentially growing photos, large-scale content-based face image retrieval is an enabling technology for many emerging applications. In this work, we aim to utilize automatically detected human attributes that contain semantic cues of the face photos to improve content-based face retrieval by constructing semantic codewords for efficient large-scale face retrieval. By leveraging human attributes in a scalable and systematic framework, we propose two orthogonal methods named attribute-enhanced sparse coding and attribute-embedded inverted indexing to improve the face retrieval in the offline and online stages. We investigate the effectiveness of different attributes and vital factors essential for face retrieval. Experimenting on two public datasets, the results show that the proposed methods can achieve up to 43.5% relative improvement in MAP compared to the existing methods.

Structural image retrieval using automatic image annotation and region based inverted file

Image retrieval has lagged far behind text retrieval despite more than two decades of intensive research effort. Most of the research on image retrieval in the last two decades are on content based image retrieval or image retrieval based on low level features. Recent research in this area focuses on semantic image retrieval using automatic image annotation. Most semantic image retrieval techniques in literature, however, treat an image as a bag of features/words while ignore the structural or spatial information in the image. In this paper, we propose a structural image retrieval method based on automatic image annotation and region based inverted file. In the proposed system, regions in an image are treated the same way as keywords in a structural text document, semantic concepts are learnt from image data to label image regions as keywords and weight is assigned to each keyword according to spatial position and relationship. As the result, images are indexed and retrieved in the same way as structural document retrieval. Specifically, images are broken down to regions which are represented using colour, texture and shape features. Region features are then quantized to create visual dictionaries which are similar to monolingual dictionaries like English or Chinese dictionaries. In the next step, a semantic dictionary similar to a bilingual dictionary like the English–Chinese dictionary is learnt to mapping image regions to semantic concepts. Finally, images are then indexed and retrieved using a novel region based inverted file data structure. Results show the proposed method has significant advantage over the widely used Bayesian annotation models.

Fast T-overlap query algorithms using graphics processor units and its applications in web data query

Given a collection of sets and a query set, a T-Overlap query identifies all sets having at least T common elements with the query. T-Overlap query is the foundation of set similarity query and join and plays an important role on web data query and processing, such as the behavior analysis of web users and the near duplicated detection of web documents. To address T-Overlap query efficiently, unlike traditional algorithms based on CPU, we aim at designing efficient GPU based algorithms. We firstly design inverted index in GPU, then choose ScanCount, a straightforward but efficient T-Overlap algorithm, as underlying algorithm to develop our GPU based T-Overlap algorithms. Depending on queries processed serially or in parallel, three new efficient algorithms are proposed based on our GPU based inverted index. Among all these three algorithms, GS-Parallel-Group processes a group of queries in parallel and supports a high degree of parallelism. Extensive experiments are carried out to compare our GPU based algorithms with other state-of-the-art CPU based algorithms. Results show that GS-Parallel-Group outperforms CPU based algorithms significantly.

Spatial Approximate String Search

This work deals with the approximate string search in large spatial databases. Specifically, we investigate range queries augmented with a string similarity search predicate in both euclidean space and road networks. We dub this query the spatial approximate string (SAS) query. In euclidean space, we propose an approximate solution, the MHR-tree, which embeds min-wise signatures into an R-tree. The min-wise signature for an index node u keeps a concise representation of the union of q-grams from strings under the subtree of u. We analyze the pruning functionality of such signatures based on the set resemblance between the query string and the q-grams from the subtrees of index nodes. We also discuss how to estimate the selectivity of a SAS query in euclidean space, for which we present a novel adaptive algorithm to find balanced partitions using both the spatial and string information stored in the tree. For queries on road networks, we propose a novel exact method, RSASSOL, which significantly outperforms the baseline algorithm in practice. The RSASSOL combines the q-gram-based inverted lists and the reference nodes based pruning. Extensive experiments on large real data sets demonstrate the efficiency and effectiveness of our approaches.

Interactive Multimodal Visual Search on Mobile Device

This paper describes a novel multimodal interactive image search system on mobile devices. The system, the Joint search with ImaGe, Speech, And Word Plus (JIGSAW ${+}$ ), takes full advantage of the multimodal input and natural user interactions of mobile devices. It is designed for users who already have pictures in their minds but have no precise descriptions or names to address them. By describing it using speech and then refining the recognized query by interactively composing a visual query using exemplary images, the user can easily find the desired images through a few natural multimodal interactions with his/her mobile device. Compared with our previous work JIGSAW, the algorithm has been significantly improved in three aspects: 1) segmentation-based image representation is adopted to remove the artificial block partitions; 2) relative position checking replaces the fixed position penalty; and 3) inverted index is constructed instead of brute force matching. The proposed JIGSAW ${+}$ is able to achieve 5% gain in terms of search performance and is ten times faster.

External-Memory Multimaps

Many data structures support dictionaries, also known as maps or associative arrays, which store and manage a set of key-value pairs. A multimap is generalization that allows multiple values to be associated with the same key. For example, the inverted file data structure that is used prevalently in the infrastructure supporting search engines is a type of multimap, where words are used as keys and document pointers are used as values. We study the multimap abstract data type and how it can be implemented efficiently online in external memory frameworks, with constant expected I/O performance. The key technique used to achieve our results is a combination of cuckoo hashing using buckets that hold multiple items with a multiqueue implementation to cope with varying numbers of values per key. Our external-memory results are for the standard two-level memory model.