Distributed Information Retrieval System Research Articles

A pipelined architecture for distributed text query evaluation

Two principal query-evaluation methodologies have been described for cluster-based implementation of distributed information retrieval systems: document partitioning and term partitioning. In a document-partitioned system, each of the processors hosts a subset of the documents in the collection, and executes every query against its local sub-collection. In a term-partitioned system, each of the processors hosts a subset of the inverted lists that make up the index of the collection, and serves them to a central machine as they are required for query evaluation. In this paper we introduce a pipelined query-evaluation methodology, based on a term-partitioned index, in which partially evaluated queries are passed amongst the set of processors that host the query terms. This arrangement retains the disk read benefits of term partitioning, but more effectively shares the computational load. We compare the three methodologies experimentally, and show that term distribution is inefficient and scales poorly. The new pipelined approach offers efficient memory utilization and efficient use of disk accesses, but suffers from problems with load balancing between nodes. Until these problems are resolved, document partitioning remains the preferred method.

Performance analysis of distributed information retrieval architectures using an improved network simulation model

The increasing number of documents that have to be indexed in different environments, particularly on the Web, and the lack of scalability of a single centralised index lead to the use of distributed information retrieval systems to effectively search for and locate the required information. In this study, we present several improvements over the two main bottlenecks in a distributed information retrieval system (the network and the brokers). We extend a simulation network model in order to represent a switched network. The new simulation model is validated by comparing the estimated response times with those obtained using a real system. We show that the use of a switched network reduces the saturation of the interconnection network, especially in a replicated system, and some improvements may be achieved using multicast messages and faster connections with the brokers. We also demonstrate that reducing the partial results sets will improve the response time of a distributed system by 53%, with a negligible probability of changing the system’s precision and recall values. Finally, we present a simple hierarchical distributed broker model that will reduce the response times for a distributed system by 55%.

Does pseudo-relevance feedback improve distributed information retrieval systems?

This paper presents a thorough analysis of the capabilities of the pseudo-relevance feedback (PRF) technique applied to distributed information retrieval (DIR). Previous studies have researched the application of PRF to improve the selection process of the best set of collections from a ranked list. This work emphasizes the effectiveness of PRF applied to the collection fusion problem. Usually, DIR systems apply PRF in the same way as traditional Information Retrieval systems. For each collection, local results are improved through PRF. A first question which arises is whether this local improvement is preserved in the final result. In addition, DIR systems merge the documents of rankings that are returned from a set of collections. Since a new global list of documents is available, we could use that list to apply PRF again, but on global level rather than on a local level. In order to apply global PRF, we have developed a merging approach called two-step RSV. Finally, we describe a number of experiments involving the two levels, local and global, of application of the PRF techniques.

The FedLemur project: Federated search in the real world

AbstractFederated search and distributed information retrieval systems provide a single user interface for searching multiple full‐text search engines. They have been an active area of research for more than a decade, but in spite of their success as a research topic, they are still rare in operational environments. This article discusses a prototype federated search system developed for the U.S. government's FedStats Web portal, and the issues addressed in adapting research solutions to this operational environment. A series of experiments explore how well prior research results, parameter settings, and heuristics apply in the FedStats environment. The article concludes with a set of lessons learned from this technology transfer effort, including observations about search engine quality in the “real world.”

순차적 전달방식(STM)을 이용한 분산정보검색시스템의 설계 및 알고리즘 구현

분산검색시스템은 중재자 또는 메타검색엔진에 의한 중앙통제로 인하여 중앙 시스템에 트래픽이 집중된다. 또한 중앙통제를 위한 복잡한 알고리즘의 설계와 별도의 하드웨어 설치 등의 비용 증가 문제가 발생한다. 따라서 이러한 문제를 해결하기 위해 여러 검색시스템이 서로 독립적인 검색기능을 가지면서도 중앙의 통제를 받지 않으면서 상호 협조할 수 있는 방식이 필요하다. 본 논문에서는 분산검색시스템과 관련된 여러 연구를 살펴보고 다중의 정보검색시스템이 중앙의 통제를 받지 않는 순차적 전달방식(STM)을 이용한 분산정보검색시스템의 Framework를 설계하며 이와 관련된 알고리즘을 구현한다 이를 위해 먼저 웹을 논리적으로 분할하여 관리하는 웹 분할 정책을 제시하며, 검색시스템의 수에 따른 순차적인 질의 처리방식을 각각 예제를 통해 설명한다. Framework에 적합한 정보검색시스템의 3 Layer구조와 각 Layer별 기능과 모듈을 또한 제시한다. 마지막으로, STM 알고리즘의 효율적 구현을 위하여 모듈 구조를 분석하고 이의 pseudocode 표현을 제시하며 서버간 순차적 질의 전달 시연과정을 통해 제안한 SIM 알고리즘이 효율적으로 동작함을 보여준다. The distributed Information Retrieval System centrally controlled by mediator or meta search engine result in congestion of heavy traffic and int he problem of increment of cost for the reason of the design of complicated algorithm for central control and installation of hardware. So to figure out this problem, the way is needed that has independent retrieval functionality and can cooperate each other without dependency. In this paper, we overview a few works involved in distributed information retrieval system, then, implement algorithm and design the frame-work of distributed information retrieval system using sequential transferring method(STM) including multiple information retrieval system separated from central control. For this first of all, we present a web partition policy which devide and manage web logically and we present the sequential query processing way by means of illustration through changing numbered information retrieval system. Then, we also present 3-layered structure of framework and function and module of each layer suitable for information retrieval system. Last of ail, for effective implementation of STM algorithm we analysis module structure and present description of pseudocode of this, and show that the proposed STM algorithm works smoothly by demonstration of sequential query transfer process between servers.

A case study of distributed information retrieval architectures to index one terabyte of text

The increasing number of documents to be indexed in many environments (Web, intranets, digital libraries) and the limitations of a single centralised index (lack of scalability, server overloading and failures), lead to the use of distributed information retrieval systems to efficiently search and locate the desired information. This work is a case study of different architectures for a distributed information retrieval system, in order to provide a guide to approximate the optimal architecture with a specific set of resources. We analyse the effectiveness of a distributed, replicated and clustered architecture simulating a variable number of workstations (from 1 up to 4096). A collection of approximately 94 million documents and 1 terabyte (TB) of text is used to test the performance of the different architectures. In a purely distributed information retrieval system, the brokers become the bottleneck due to the high number of local answer sets to be sorted. In a replicated system, the network is the bottleneck due to the high number of query servers and the continuous data interchange with the brokers. Finally, we demonstrate that a clustered system will outperform a replicated system if a high number of query servers is used, essentially due to the reduction of the network load. However a change in the distribution of the users’ queries could reduce the performance of a clustered system.

Using Coloured Petri Nets to Compare Mobile Agent Design Patterns

Concurrent distributed systems modelling is a complex task. When we add mobility, the scenario is still worse, as specific problems are added. Petri nets is a largely used formalism to model and analyze complex systems, specially distributed and concurrent ones. Petri nets models are executable, allowing the simulation of abstract specifications. For this reason, Petri nets have been widely used as a validation and specification tool. This paper presents results of a performance comparative study of three mobile agents design patterns. The Itinerary, Star-Shaped and Branching migration patterns were investigated as solutions to a distributed information retrieval system. The three solutions were modelled with timed Coloured Petri Nets. The models were validated using simulation and occurrence graph analysis techniques. Experiments were conducted using further simulations and data were collected in order to evaluate the performance of each design pattern. The obtained data allow us to conclude that, for the studied scenarios, the Branching Pattern presents the the best performance.

Using Coloured Petri Nets to Compare Mobile Agent Design Patterns

Concurrent distributed systems modelling is a complex task. When we add mobility, the scenario is still worse, as specific problems are added. Petri nets is a largely used formalism to model and analyze complex systems, specially distributed and concurrent ones. Petri nets models are executable, allowing the simulation of abstract specifications. For this reason, Petri nets have been widely used as a validation and specification tool. This paper presents results of a performance comparative study of three mobile agents design patterns. The Itinerary, Star-Shaped and Branching migration patterns were investigated as solutions to a distributed information retrieval system. The three solutions were modelled with timed Coloured Petri Nets. The models were validated using simulation and occurrence graph analysis techniques. Experiments were conducted using further simulations and data were collected in order to evaluate the performance of each design pattern. The obtained data allow us to conclude that, for the studied scenarios, the Branching Pattern presents the the best performance.

Optimization in the migration problem of mobile agents in distributed information retrieval systems

In this paper, we employ genetic algorithms to solve the migration problem (MP). We propose a new encoding scheme to represent trees, which is composed of two parts: the pre-ordered traversal sequence of tree vertices and the children number sequence of corresponding tree vertices. The proposed encoding scheme has the advantages of simplicity for encoding and decoding, ease for GA operations, and better equilibrium between exploration and exploitation. It is also adaptive in that, with few restrictions on the length of code, it can be freely lengthened or shortened according to the characteristics of the problem space. Furthermore, the encoding scheme is highly applicable to the degree-constrained minimum spanning tree problem because it also contains the degree information of each node. The simulation results demonstrate the higher performance of our algorithm, with fast convergence to the optima or sub-optima on various problem sizes. Comparing with the binary string encoding of vertices, when the problem size is large, our algorithm runs remarkably faster with comparable search capability.

Information retrieval with distributed databases: analytic models of performance

The major emphasis is on analytical techniques for predicting the performance of various collection fusion scenarios. Knowledge of analytical models of information retrieval system performance, both with single processors and with multiple processors, increases our understanding of the parameters (e.g., number of documents, ranking algorithms, stemming algorithms, stop word lists, etc.) affecting system behavior. While there is a growing literature on the implementation of distributed information retrieval systems and digital libraries, little research has focused on analytic models of performance. We analytically describe the performance for single and multiple processors, both when different processors have the same parameter values and when they have different values. The use of different ranking algorithms and parameter values at different sites is examined.

Partial Collection Replication for Information Retrieval

The explosion of content in distributed information retrieval (IR) systems requires new mechanisms in order to attain timely and accurate retrieval of unstructured text. This paper shows how to exploit locality by building, using, and searching partial replicas of text collections in a distributed IR system. In this work, a partial replica includes a subset of the documents from larger collection(s) and the corresponding inference network search mechanism. For each query, the distributed system determines if partial replica is a good match and then searches it, or it searches the original collection. We demonstrate the scenarios where partial replication performs better than systems that use caches which only store previous query and answer pairs. We first use logs from THOMAS and Excite to examine query locality using query similarity versus exact match. We show that searching replicas can improve locality (from 3 to 19%) over the exact match required by caching. Replicas increase locality because they satisfy queries which are distinct but return the same or very similar answers. We then present a novel inference network replica selection function. We vary its parameters and compare it to previous collection selection functions, demonstrating a configuration that directs most of the appropriate queries to replicas in a replica hierarchy. We then explore the performance of partial replication in a distributed IR system. We compare it with caching and partitioning. Our validated simulator shows that the increases in locality due to replication make it preferable to caching alone, and that even a small increase of 4% in locality translates into a performance advantage. We also show a hybrid system with caches and replicas that performs better than each on their own.

Evaluating the performance of distributed architectures for information retrieval using a variety of workloads

The information explosion across the Internet and elswhere offers access to an increasing number of document collections. In order for users to effectively access these collections, information retrieval (IR) systems must provide coordinated, concurrent, and distributed access. In this article, we explore how to achieve scalable performance in a distributed system for collection sizes ranging from 1GB to 128GB. We implement a fully functional distributed IR system based on a multithreaded version of the Inquery simulation model. We measure performance as a function of system parameters such as client command rate, number of document collections, ter ms per query, query term frequency, number of answers returned, and command mixture. Our results show that it is important to model both query and document commands because the heterogeneity of commands significantly impacts performance. Based on our results, we recommend simple changes to the prototype and evaluate the changes using the simulator. Because of the significant resource demands of information retrieval, it is not difficult to generate workloads that overwhelm system resources regardless of the architecture. However under some realistic workloads, we demonstrate system organizations for which response time gracefully degrades as the workload increases and performance scales with the number of processors. This scalable architecture includes a surprisingly small number of brokers through which a large number of clients and servers communicate.

Impact of the Internet/World-Wide Web on today's society with reference to business

The juxtaposition of the words Internet/World-Wide Web in the title of this article implies equivalence between the Internet and the Web. What distinguishes the Web from the Internet? The Webster Dictionary (2000) defines the World-Wide Web as 'a part of the Internet designed to allow easier navigation of the network through the use of graphical user interfaces and hypertext links between different addresses [called also Web]'. The Webster definition of the Internet is 'an electronic communications network that connects computer networks and organizational computer facilities around the world'. The Instantweb Computing Dictionary(2000) defines the Web as: 'An Internet client-server hypertext distributed information retrieval system ... On the WWW everything (documents, menus, indices) is represented to the user as a hypertext object in HTML format. Hypertext links refer to other documents by their URLs. These can refer to local or remote resources accessible via FTP, Gopher, Telnet or news, as well as those available via the http protocol used to transfer hypertext documents'. The Instantweb Dictionary defines internet as: 'Any set of networks interconnected with routers. The Internet is the biggest example of an internet.'

Models of a distributed information retrieval system based on thesauri with weights

problem of combining n local information systems S 1, S 2, …, S n based on thesauri with weights in one system S is considered. The distributed information retrieval system S may be assumed to be a network of the local systems S j . Both the generalization relation on the set of descriptors and weights of descriptors in descriptions of documents are taken into account. The fundamental properties of distributed systems are described. The inverted file structure is often used to organize data in the information retrieval system. Operations on inverted lists are modified in order to use them in the distributed information system. While retrieving the response to any query in a distributed system, we may use the existing inverted lists from local subsystems. In a distributed system the retrieval process follows almost in the same way as in the method of inverted files in conventional systems. It differs only due to use of the additional union operations on selected inverted lists from any local subsystems. The presented retrieval rules may be viewed as the logical approach in implementing a physical distributed information retrieval system.

Prototyping a distributed information retrieval system that uses statistical ranking

Centralized systems continue to dominate the information retrieval market, with increased competition from CD-ROM based systems. As more large organizations begin to implement office automation systems, however, many will find that neither of these types of retrieval systems will satisfy their requirements, especially those requirements involving easy integration into other systems and heavy usage by casual end users. A prototype distributed information retrieval system was designed and built using a distributed architecture and using statistical ranking techniques to help provide better service for the end user. The distributed architecture was shown to be a feasible alternative to centralized or CD-ROM information retrieval, and user testing of the ranking methodology showed both widespread user enthusiasm for this retrieval technique and very fast response times (on the order of one second for 300 megabytes of data).

Strategies for building distributed information retrieval systems

In this article we discuss the need for distributed information retrieval Systems. A number of possible configurations are presented. A general approach to the design of such systems is discussed. A prototype implementation is described together with the experiences gained from this implementation.

Organization of the inverted files in a distributed information retrieval system based on thesauri

This paper describes how the operations on the local inverted files are to be modified in order to use them in the distributed information retrieval system based on thesauri. The global system consists of n local retrieval systems. The presented retrieval rules may be viewed as the logical approach in implementing a physical distributed retrieval system.

On a model of distributed information retrieval systems based on thesauri

The properties of a global model consisting of n local information retrieval systems based on thesaurus are investigated.