Distributed Information Retrieval Research Articles

A timed mobile agent planning approach for distributed information retrieval in dynamic network environments

The number of mobile agents and total execution time are two factors used to represent the system overhead that must be considered as part of mobile agent planning (MAP) for distributed information retrieval. In addition to these two factors, the time constraints at the nodes of an information repository must also be taken into account when attempting to improve the quality of information retrieval. In previous studies, MAP approaches could not consider dynamic network conditions, e.g., variable network bandwidth and disconnection, such as are found in peer-to-peer (P2P) computing. For better performance, mobile agents that are more sensitive to network conditions must be used. In this paper, we propose a new MAP approach that we have named Timed Mobile Agent Planning (Tmap). The proposed approach minimizes the number of mobile agents and total execution time while keeping the turnaround time to a minimum, even if some nodes have a time constraint. It also considers dynamic network conditions to reflect the dynamic network condition more accurately. Moreover, we incorporate a security and fault-tolerance mechanism into the planning approach to better adapt it to real network environments.

Testing the cluster hypothesis in distributed information retrieval

How to merge and organise query results retrieved from different resources is one of the key issues in distributed information retrieval. Some previous research and experiments suggest that cluster-based document browsing is more effective than a single merged list. Cluster-based retrieval results presentation is based on the cluster hypothesis, which states that documents that cluster together have a similar relevance to a given query. However, while this hypothesis has been demonstrated to hold in classical information retrieval environments, it has never been fully tested in heterogeneous distributed information retrieval environments. Heterogeneous document representations, the presence of document duplicates, and disparate qualities of retrieval results, are major features of an heterogeneous distributed information retrieval environment that might disrupt the effectiveness of the cluster hypothesis. In this paper we report on an experimental investigation into the validity and effectiveness of the cluster hypothesis in highly heterogeneous distributed information retrieval environments. The results show that although clustering is affected by different retrieval results representations and quality, the cluster hypothesis still holds and that generating hierarchical clusters in highly heterogeneous distributed information retrieval environments is still a very effective way of presenting retrieval results to users.

The FedLemur project: Federated search in the real world

AbstractFederated search and distributed information retrieval systems provide a single user interface for searching multiple full‐text search engines. They have been an active area of research for more than a decade, but in spite of their success as a research topic, they are still rare in operational environments. This article discusses a prototype federated search system developed for the U.S. government's FedStats Web portal, and the issues addressed in adapting research solutions to this operational environment. A series of experiments explore how well prior research results, parameter settings, and heuristics apply in the FedStats environment. The article concludes with a set of lessons learned from this technology transfer effort, including observations about search engine quality in the “real world.”

Information source selection for resource constrained environments

Distributed information retrieval has pressing scalability concerns due to the growing number of independent sources of on-line data and the emerging applications. A promising solution to distributed retrieval is metasearching, which dispatches a user's query to multiple sources and gathers the results into a single result set. An important component of metasearching is selecting the set of information sources most likely to provide relevant documents. Recent research has focused on how to obtain statistics for the selection task. In this paper we discuss different information source selection approaches and their applicability for resource-constrained sensor network applications.

Opportunistic Carrier Sensing for Energy-Efficient Information Retrieval in Sensor Networks

We consider distributed information retrieval for sensor networks with cluster heads or mobile access points. The performance metric used in the design is energy efficiency defined as the ratio of the average number of bits reliably retrieved by the access point to the total amount of energy consumed. A distributed opportunistic transmission protocol is proposed using a combination of carrier sensing and backoff strategy that incorporates channel state information (CSI) of individual sensors. By selecting a set of sensors with the best channel states to transmit, the proposed protocol achieves the upper bound on energy efficiency when the signal propagation delay is negligible. For networks with substantial propagation delays, a backoff function optimized for energy efficiency is proposed. The design of this backoff function utilizes properties of extreme statistics and is shown to have mild performance loss in practical scenarios. We also demonstrate that opportunistic strategies that use CSI may not be optimal when channel acquisition at individual sensors consumes substantial energy. We show further that there is an optimal sensor density for which the opportunistic information retrieval is the most energy efficient. This observation leads to the design of the optimal sensor duty cycle.

Biological Information Standards

Biological Information Standards

순차적 전달방식(STM)을 이용한 분산정보검색시스템의 설계 및 알고리즘 구현

분산검색시스템은 중재자 또는 메타검색엔진에 의한 중앙통제로 인하여 중앙 시스템에 트래픽이 집중된다. 또한 중앙통제를 위한 복잡한 알고리즘의 설계와 별도의 하드웨어 설치 등의 비용 증가 문제가 발생한다. 따라서 이러한 문제를 해결하기 위해 여러 검색시스템이 서로 독립적인 검색기능을 가지면서도 중앙의 통제를 받지 않으면서 상호 협조할 수 있는 방식이 필요하다. 본 논문에서는 분산검색시스템과 관련된 여러 연구를 살펴보고 다중의 정보검색시스템이 중앙의 통제를 받지 않는 순차적 전달방식(STM)을 이용한 분산정보검색시스템의 Framework를 설계하며 이와 관련된 알고리즘을 구현한다 이를 위해 먼저 웹을 논리적으로 분할하여 관리하는 웹 분할 정책을 제시하며, 검색시스템의 수에 따른 순차적인 질의 처리방식을 각각 예제를 통해 설명한다. Framework에 적합한 정보검색시스템의 3 Layer구조와 각 Layer별 기능과 모듈을 또한 제시한다. 마지막으로, STM 알고리즘의 효율적 구현을 위하여 모듈 구조를 분석하고 이의 pseudocode 표현을 제시하며 서버간 순차적 질의 전달 시연과정을 통해 제안한 SIM 알고리즘이 효율적으로 동작함을 보여준다. The distributed Information Retrieval System centrally controlled by mediator or meta search engine result in congestion of heavy traffic and int he problem of increment of cost for the reason of the design of complicated algorithm for central control and installation of hardware. So to figure out this problem, the way is needed that has independent retrieval functionality and can cooperate each other without dependency. In this paper, we overview a few works involved in distributed information retrieval system, then, implement algorithm and design the frame-work of distributed information retrieval system using sequential transferring method(STM) including multiple information retrieval system separated from central control. For this first of all, we present a web partition policy which devide and manage web logically and we present the sequential query processing way by means of illustration through changing numbered information retrieval system. Then, we also present 3-layered structure of framework and function and module of each layer suitable for information retrieval system. Last of ail, for effective implementation of STM algorithm we analysis module structure and present description of pseudocode of this, and show that the proposed STM algorithm works smoothly by demonstration of sequential query transfer process between servers.

A case study of distributed information retrieval architectures to index one terabyte of text

The increasing number of documents to be indexed in many environments (Web, intranets, digital libraries) and the limitations of a single centralised index (lack of scalability, server overloading and failures), lead to the use of distributed information retrieval systems to efficiently search and locate the desired information. This work is a case study of different architectures for a distributed information retrieval system, in order to provide a guide to approximate the optimal architecture with a specific set of resources. We analyse the effectiveness of a distributed, replicated and clustered architecture simulating a variable number of workstations (from 1 up to 4096). A collection of approximately 94 million documents and 1 terabyte (TB) of text is used to test the performance of the different architectures. In a purely distributed information retrieval system, the brokers become the bottleneck due to the high number of local answer sets to be sorted. In a replicated system, the network is the bottleneck due to the high number of query servers and the continuous data interchange with the brokers. Finally, we demonstrate that a clustered system will outperform a replicated system if a high number of query servers is used, essentially due to the reduction of the network load. However a change in the distribution of the users’ queries could reduce the performance of a clustered system.

Distributed content-based visual information retrieval system on peer-to-peer networks

With the recent advances of distributed computing, the limitation of information retrieval from a centralized image collection can be removed by allowing distributed image data sources to interact with each other for data storage sharing and information retrieval. In this article, we present our design and implementation of DISCOVIR: DIStributed COntent-based Visual Information Retrieval system using the Peer-to-Peer (P2P) Network. We describe the system architecture and detail the interactions among various system modules. Specifically, we propose a Firework Query Model for distributed information retrieval, which aims to reduce the network traffic of query passing in the network. We carry out experiments to show the distributed image retrieval system and the Firework information retrieval algorithm. The results show that the algorithm reduces network traffic while increases searching performance.

CEMAP: COST-EFFECTIVE MOBILE AGENT PLANNING

Two significant performance factors in Mobile Agent Planning (MAP) for distributed information retrieval are the number of mobile agents and the total execution time. Using fewer mobile agents results in less network traffic and consumes less bandwidth. Regardless of the number of agents used, the total execution time for a task must be kept to a minimum. A retrieval service must minimize both these factors for better system performance, and at the same time, it must be able to supply the required information to users as quickly as possible. In this paper, we propose heuristic algorithms, called Cost-Effective MAP (CEMAP), to minimize both the number of mobile agents and the total execution time under the condition that the turnaround time is kept to a minimum. Although these algorithms tend to slightly increase the planning cost, a simulation study shows that these algorithms enhance the system performance significantly. By adopting these algorithms, systems can maintain lower network traffic while satisfying the minimum turnaround time.

Using Coloured Petri Nets to Compare Mobile Agent Design Patterns

Concurrent distributed systems modelling is a complex task. When we add mobility, the scenario is still worse, as specific problems are added. Petri nets is a largely used formalism to model and analyze complex systems, specially distributed and concurrent ones. Petri nets models are executable, allowing the simulation of abstract specifications. For this reason, Petri nets have been widely used as a validation and specification tool. This paper presents results of a performance comparative study of three mobile agents design patterns. The Itinerary, Star-Shaped and Branching migration patterns were investigated as solutions to a distributed information retrieval system. The three solutions were modelled with timed Coloured Petri Nets. The models were validated using simulation and occurrence graph analysis techniques. Experiments were conducted using further simulations and data were collected in order to evaluate the performance of each design pattern. The obtained data allow us to conclude that, for the studied scenarios, the Branching Pattern presents the the best performance.

Using Coloured Petri Nets to Compare Mobile Agent Design Patterns

Concurrent distributed systems modelling is a complex task. When we add mobility, the scenario is still worse, as specific problems are added. Petri nets is a largely used formalism to model and analyze complex systems, specially distributed and concurrent ones. Petri nets models are executable, allowing the simulation of abstract specifications. For this reason, Petri nets have been widely used as a validation and specification tool. This paper presents results of a performance comparative study of three mobile agents design patterns. The Itinerary, Star-Shaped and Branching migration patterns were investigated as solutions to a distributed information retrieval system. The three solutions were modelled with timed Coloured Petri Nets. The models were validated using simulation and occurrence graph analysis techniques. Experiments were conducted using further simulations and data were collected in order to evaluate the performance of each design pattern. The obtained data allow us to conclude that, for the studied scenarios, the Branching Pattern presents the the best performance.

Optimization in the migration problem of mobile agents in distributed information retrieval systems

In this paper, we employ genetic algorithms to solve the migration problem (MP). We propose a new encoding scheme to represent trees, which is composed of two parts: the pre-ordered traversal sequence of tree vertices and the children number sequence of corresponding tree vertices. The proposed encoding scheme has the advantages of simplicity for encoding and decoding, ease for GA operations, and better equilibrium between exploration and exploitation. It is also adaptive in that, with few restrictions on the length of code, it can be freely lengthened or shortened according to the characteristics of the problem space. Furthermore, the encoding scheme is highly applicable to the degree-constrained minimum spanning tree problem because it also contains the degree information of each node. The simulation results demonstrate the higher performance of our algorithm, with fast convergence to the optima or sub-optima on various problem sizes. Comparing with the binary string encoding of vertices, when the problem size is large, our algorithm runs remarkably faster with comparable search capability.

Information retrieval with distributed databases: analytic models of performance

The major emphasis is on analytical techniques for predicting the performance of various collection fusion scenarios. Knowledge of analytical models of information retrieval system performance, both with single processors and with multiple processors, increases our understanding of the parameters (e.g., number of documents, ranking algorithms, stemming algorithms, stop word lists, etc.) affecting system behavior. While there is a growing literature on the implementation of distributed information retrieval systems and digital libraries, little research has focused on analytic models of performance. We analytically describe the performance for single and multiple processors, both when different processors have the same parameter values and when they have different values. The use of different ranking algorithms and parameter values at different sites is examined.

A semisupervised learning method to merge search engine results

The proliferation of searchable text databases on local area networks and the Internet causes the problem of finding information that may be distributed among many disjoint text databases ( distributed information retrieval ). How to merge the results returned by selected databases is an important subproblem of the distributed information retrieval task. Previous research assumed that either resource providers cooperate to provide normalizing statistics or search clients download all retrieved documents and compute normalized scores without cooperation from resource providers.This article presents a semisupervised learning solution to the result merging problem. The key contribution is the observation that information used to create resource descriptions for resource selection can also be used to create a centralized sample database to guide the normalization of document scores returned by different databases. At retrieval time, the query is sent to the selected databases, which return database-specific document scores, and to a centralized sample database , which returns database-independent document scores. Documents that have both a database-specific score and a database-independent score serve as training data for learning to normalize the scores of other documents. An extensive set of experiments demonstrates that this method is more effective than the well-known CORI result-merging algorithm under a variety of conditions.

Soft approaches to distributed information retrieval

In this paper we consider the context of information retrieval in a network. Two distinct situations of distributed retrieval are considered: the case in which we have distinct databases residing on distinct servers having their own IRSs, and the case in which we have a unique, huge and distributed information repository (like the WWW), and distinct IRSs (search engines), which can be used to inquiry the same collection. In both situations we have the problem of merging the results produced by the distinct IRS into a unique ordered list, which constitutes the unified answer to the user query. In solving the problem of list fusion two main issues are involved. The first one concerns the decision of how many retrieved documents to select from each individual documents list. The second aspect is to define a procedure to fuse the individual lists of retrieved documents into an overall ordered list. In this paper some fuzzy approaches to solve the two above-mentioned problems are presented.

DISTRIBUTED INFORMATION RETRIEVAL: A MULTI-OBJECTIVE RESOURCE SELECTION APPROACH

Information retrieval is becoming increasingly concerned with resource selection and data fusion for distributed archives. In distributed information retrieval, a user submits a query to a broker, which determines a solution for how to yield a given number of documents from all available resources. In this paper, we present a multi-objective model for resource selection, in which four aspects: a document's relevance to the given query, time, monetary cost, and the chance of getting document duplicates from resources, are considered simultaneously. Some variants of this multi-objective model, aimed at achieving better implementation efficiency, are also proposed.

SIGIR 2003 workshop on distributed information retrieval

During the last decade companies, governments, and research groups worldwide have directed significant effort towards the creation of sophisticated digital libraries across a variety of disciplines. As digital libraries proliferate, in a variety of media, and from a variety of sources, problems of resource selection and data fusion become major obstacles. Traditional search engines, even very large systems such as Google, are unable to provide access to the “Hidden Web” of information that is only available via digital library search interfaces. Effective, reliable information retrieval also requires the ability to pose multimedia queries across many digital libraries. The answer to a query about the lyrics to a folk song might be text or an audio recording, but few systems today could deliver both data types in response to a single, simple query. Distributed Information Retrieval addresses issues that arise when people have routine access to thousands of multimedia digital libraries.

Result merging strategies for a current news metasearcher

Metasearching of online current news services is a potentially useful Web application of distributed information retrieval techniques. We constructed a realistic current news test collection using the results obtained from 15 current news Web sites (including ABC News, BBC and AllAfrica) in response to 107 topical queries. Results were judged for relevance by independent assessors. Online news services varied considerably both in the usefulness of the results sets they returned and also in the amount of information they provided which could be exploited by a metasearcher. Using the current news test collection we compared a range of different merging methods. We found that a low-cost merging scheme based on a combination of available evidence (title, summary, rank and server usefulness) worked almost as well as merging based on downloading and rescoring the actual news articles.

D-Agent: an approach to mobile agent planning for distributed information retrieval

Owing to advances in home networking, Internet, and mobile computing technology, the retrieval of information that is required in a timely manner has increased in importance for both the home and business user. This type of information can usually be delivered to users within the required time. Therefore, a retrieval service needs to be able to supply the required information to users within a specified time given by the user, and at the same time, it needs to have minimum system overheads from network traffic when performing the retrieval task for an anticipated increasing number of users. In this paper, we propose a new agent-planning algorithm, called d-Agent, for distributed information retrieval. This planning algorithm has two goals: (1) to guarantee a given turn-around time; (2) to schedule mobile agents (the number of agents and each agent's itinerary) optimally while maintaining low system overheads. Although the algorithm tends to slightly increase the planning cost because of these two requirements, a simulation study shows that our algorithm is practical and realistic, and can be directly applied to distributed information retrieval with deadline constraints.