Distributed Information Retrieval System Research Articles

A Scalable Real-Time Agent-Based Information Retrieval Engine

In distributed information retrieval systems, information in web should be ranked based on a combination of multiple features. Linear combination of ranks has been the dominant approach due to its simplicity and efficiency. Such a combination scheme in distributed infrastructure requires that ranks in resources or agents are comparable to each other. The main challenge is how to transform the raw rank values of different criteria appropriately to make them comparable before any combination. In this manuscript, we will demonstrate how to rank Web documents based on its resource-provided information stream and how to combine and incorporate several raking schemas in one time. The system was tested on the queries provided by a Text Retrieval Conference (TREC), and our experimental results showed that it is robust and efficient compared with similar platforms that used offline data resources.

A New Approach of Intelligent Data Retrieval Paradigm

What is a real time agent, how does it remedy ongoing daily frustrations for users, and how does it improve the retrieval performance in World Wide Web? These are the main question we focus on this manuscript. In many distributed information retrieval systems, information in agents should be ranked based on a combination of multiple criteria. Linear combination of ranks has been the dominant approach due to its simplicity and effectiveness. Such a combination scheme in distributed infrastructure requires that the ranks in resources or agents are comparable to each other before combined. The main challenge is transforming the raw rank values of different criteria appropriately to make them comparable before any combination. Different ways for ranking agents make this strategy difficult. In this research, we will demonstrate how to rank Web documents based on resource-provided information how to combine several resources raking schemas in one time. The proposed system was implemented specifically in data provided by agents to create a comparable combination for different attributes. The proposed approach was tested on the queries provided by Text Retrieval Conference (TREC). Experimental results showed that our approach is effective and robust compared with offline search platforms.

Multi-agent-based hybrid peer-to-peer system for distributed information retrieval

With the increasingly huge amount of data located in various databases and the need for users to access them, distributed information retrieval (DIR) has been at the core of the preoccupations of a number of researchers. Indeed, numerous DIR systems and architectures have been proposed including the broker-based architecture. Moreover, providing DIR with more flexibility and adaptability has led researchers thinking to build DIR with software agents. Thus, this research proposes a design and an implementation of a novel system based on the broker-based architecture and the peer-to-peer (P2P) network called broker-based P2P network. The proposed architecture is implemented with a multi-agent system (MAS) where the main agent playing the role of the broker, receives query from a peer agent and forwards them to other peer agents each with their index and resources. Upon completing retrieval process at each peer agent, results are directly sent to the peer agent that initiated the query without using the broker agent. Java Agent DEvelopment framework (JADE) is used to implement the agents and, for experiments, TERRIER (TERabyte RetRIEveR) is extended and used as the search engine to retrieve the Text Retrieval Conference (TREC) collections dataset notably TREC-6. The peer agent that originated the query progressively collects results coming from other peer agents, normalises and merges them and then proceeds with re-ranking. For normalisation, MinMax and Sum that are unsupervised normalisation methods are used.

M67 PSYCHIATRIC DISORDERS AND SLC6A4 GENE VARIANTS: POSSIBLE MODULATION OF ALCOHOL DEPENDENCE AND ALZHEIMER'S DISEASE

The selection of information sources in a distributed information retrieval environment remains a critical issue. In this context, it is known that a distributed information retrieval system consists of a huge number of sources. Ensuring retrieval effectiveness is to search only sources which are likely to contain relevant information for a query. An important number of heuristics exist among which we quote genetic algorithm that is used to solve the above problem. The proposed genetic algorithm consists in finding the best selection in large space of potential solutions; where a solution is represented as a combination of a set of sources. The improvement of selection accuracy is assured based on the user’s track through the use of sources, to say that source description is enriched with tags from the tagging history.

Design and Implementation of Distributed Information Retrieval System Based on VOTable Protocol

Astronomy has entered a period of massive data with rapid development of the information age. Astronomical research is becoming more and more mature. Nevertheless, there is still a big problem with the sharing massive astronomical data. Because the astronomical data is widely distributed in various observatory stations around the world and the structure, format and language of astronomical data in different regions and even different countries are different, the retrieval and storage of astronomical data is facing great challenge. This paper designs and implements a distributed information retrieval system based on VOTable protocol. In this way, we can improve the sharing of massive astronomical data greatly.

Enhancing information source selection using a genetic algorithm and social tagging

The selection of information sources in a distributed information retrieval environment remains a critical issue. In this context, it is known that a distributed information retrieval system consists of a huge number of sources. Ensuring retrieval effectiveness is to search only sources which are likely to contain relevant information for a query. An important number of heuristics exist among which we quote genetic algorithm that is used to solve the above problem. The proposed genetic algorithm consists in finding the best selection in large space of potential solutions; where a solution is represented as a combination of a set of sources. The improvement of selection accuracy is assured based on the user’s track through the use of sources, to say that source description is enriched with tags from the tagging history.

Integrating social profile to improve the source selection and the result merging process in distributed information retrieval

In this paper we present a new personalized approach, which integrates a social profile into a distributed search system. Most previous studies on distributed information retrieval are based on textual information, and rarely consider any social information. Based on this observation, we propose an approach which exploits the social profile and the different relations between social entities. We believe that this method can: (i) enhance a query expansion, (ii) personalize and improve both the source selection and the result merging process in distributed information retrieval systems.

Distributed Information Retrieval: Developments and Strategies

Opposed to centralized search where Websites are crawled and indexed, Distributed Information Retrieval (DIR), also known as Federated Search, is a powerful way to comprehensively search multiple databases in real-time simultaneously. DIR is preferred to centralized search environments in a number of ways, characteristically among them are: 1. the diversity of resources that are made available; 2. improving scalability and reducing server load and network traffic; 3. the leverage of accessing the hidden or deep Web.There are three major phases/tasks of a DIR (i) resource description or collection representation (ii) resource selection and (iii) result merging. This paper aims at providing a comprehensive review on the various phases of DIR and also some current strategies being recommended in enhancing and improving the smooth implementation of a DIR system.

Test-bed building process for context-aware peer-to-peer information retrieval evaluation

With the rapid growth of information rate in the web, it is of paramount importance to develop scalable information retrieval approaches. These approaches are based on distributed information retrieval systems. Likewise, the emergence of contextual information retrieval has introduced the research personalisation which attempts to reduce the research ambiguity and to return most appropriate information to the user. Therefore, it seems a compelling issue, to implement techniques related to contextual peer-to-peer information retrieval (P2P-IR) to better exploit the personalisation of information in order to achieve a more efficient access to data. In this respect, evaluating these approaches in contextual P2P-IR is a crucial step. Indeed, to the best of our knowledge, there is no available test-beds dedicated to the evaluation of contextual P2P-IR. In particular, we propose, two methods. The first is based on existing collections that are enriched with external semantic resources. The second offers the possibility to build test-beds from multiple sources data.

Update Strategies of Global Statistics in Distributed Information Retrieval System

“Update Strategies of Global Statistics in Distributed Information Retrieval System” By Jianjun Lang, Qigang Jiang Journal of Networks Volume 9, Number 10, October 2014, Page(s): 2751-2757 After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, the above paper has been found to be in violation of Academy Publisher’s Publication Principles. The first author (Jianjun Lang) has taken full responsibility and this violation was done without the knowledge of the second author (Qigang Jiang). This paper contains significant portions of original text from the papers cited below. The original text was copied without attribution (including appropriate references to the original authors and/or paper title) and without permission: Due to the nature of this violation, reasonable effort should be made to remove all past references to this paper, and future references should be made to the following article: Lijing Wang, “The Update Strategies of Global Statistics in Distributed Information Retrieval System,” Master Thesis, University of Chinese Academy of Sciences, April 2013, Beijing, China. (In Chinese)

Design and implementation of distributed retrieval system for electronic products information

In order to obtain the useful information that can satisfy the user requirements,this paper proposed a distributed information retrieval system based on Hadoop and Lucene handling the Web electronic products information retrieval.In order to improve the retrieval efficiency,using the Map and Reduce method of Hadoop technology implemented the storage of distributed index files and using Lucene technology implemented the file access of distributed index files.At the same time,it also proposed an improved method at fine grain retrieval level,which reduced the index building time.The experiment demonstrates that our distributed information retrieval system has a good retrieval performance for Web electronic products information.

Usercentric Operational Decision Making in Distributed Information Retrieval

Information specialists in enterprises regularly use distributed information retrieval (DIR) systems that query a large number of information retrieval (IR) systems, merge the retrieved results, and display them to users. There can be considerable heterogeneity in the quality of results returned by different IR servers. Further, because different servers handle collections of different sizes and have different processing and bandwidth capacities, there can be considerable heterogeneity in their response times. The broker in the DIR system has to decide which servers to query, how long to wait for responses, and which retrieved results to display based on the benefits and costs imposed on users. The benefit of querying more servers and waiting longer is the ability to retrieve more documents. The costs may be in the form of access fees charged by IR servers or user's cost associated with waiting for the servers to respond. We formulate the broker's decision problem as a stochastic mixed-integer program and present analytical solutions for the problem. Using data gathered from FedStats—a system that queries IR engines of several U.S. federal agencies—we demonstrate that the technique can significantly increase the utility from DIR systems. Finally, simulations suggest that the technique can be applied to solve the broker's decision problem under more complex decision environments.

The 8th workshop on large-scale distributed systems for information retrieval (LSDS-IR'10)

The size of theWeb as well as user bases of search systems continue to grow exponentially. Consequently, providing subsecond query response times and high query throughput become quite challenging for large-scale information retrieval systems. Distributing different aspects of search (e.g., crawling, indexing, and query processing) is essential to achieve scalability in large-scale information retrieval systems. The 8th Workshop on Large-Scale Distributed Systems for Information Retrieval (LSDS-IR'10) has provided a venue to discuss the current research challenges and identify new directions for distributed information retrieval. The workshop contained two industry talks as well as six research paper presentations. The hot topics in this year's workshop were collection selection architectures, application of MapReduce to information retrieval problems, similarity search, geographically distributed web search, and optimization techniques for search efficiency.

7th workshop on large-scale distributed systems for information retrieval (LSDS-IR'09)

Due to the dramatically increasing amount of available data, effective and scalable solutions for data organization and search are essential. Distributed solutions naturally provide promising alternatives to standard centralized approaches. With the computational power of thousands or millions of computers in clusters or peer-to-peer systems, the challenges that arise are manifold, ranging from efficient resource discovery to issues in load balancing and distributed query processing. The 2009 edition of the Workshop on Large-Scale Distributed Systems for Information Retrieval (LSDS-IR'09) provided a forum for researchers to discuss these problems and to define new directions in research on Distributed Information Retrieval. The Workshop program featured research contributions in the areas of collection selection, similarity search, index compression, distributed indexing, query processing, distributed computing, and network formation. In addition, there were two industry talks on large scale Web search and distributed computing.

Workshop on large-scale distributed systems for information retrieval

Due to the dramatically increasing amount of available data, effective and scalable solutions for data organization and search are essential. Distributed solutions naturally provide promising alternatives to standard centralized approaches. With the computational power of thousands or millions of computers in clusters or peer-to-peer systems, the challenges that arise are manifold, ranging from efficient resource discovery to issues in load balancing and distributed query processing. The 2008 edition of the Workshop on Large-Scale Distributed Systems for Information Retrieval (LSDS-IR'08) provided a forum for researchers to discuss these problems and to define new directions for the work on Distributed Information Retrieval. The Workshop program featured research contributions in the areas of similarity search, resource selection, network organization schemes, issues of data quality, result ranking techniques and query routing algorithms.

Consensus-based evaluation framework for distributed information retrieval systems

Multi-agent systems have been attacking the challenges of information retrieval tasks on distributed environment. In this paper, we propose a consensus choice selection method based framework to evaluate the performance of cooperative information retrieval tasks of the multiple agents. Thereby, two well-known measurements, precision and recall, are extended to handle consensual closeness (i.e., local and global consensus) between the sets of retrieved results. We show that in a motivating example the proposed criteria are prone to solve the rigidity problem of classical precision and recall. More importantly, the retrieved results can be ranked with respect to the consensual score, and the ranking mechanism has been verified to be more reasonable.

User-Centric Operational Decision-Making in Distributed Information Retrieval

Information specialists in enterprises and consumers on the Internet regularly use Distributed Information Retrieval (DIR) systems that query a large number of Information Retrieval (IR) systems, merge the retrieved results and display them to users. There can be considerable heterogeneity in the quality of results returned by different IR servers. Further, since different servers handle collections of different sizes, have different processing and bandwidth capacities, there can be considerable heterogeneity in their response times. The broker in the distributed IR system thus has to decide which servers to query, how long to wait for responses and which retrieved results to display based on the benefits and costs imposed on users. The benefit of querying more servers and waiting longer is the ability to retrieve more documents. The costs may be in the form of access fees charged by IR servers or user's cost associated with waiting for the servers to respond. We formulate the broker's decision problem as a stochastic mixed integer program. We present closed-form results for the optimal query set and wait time in the special case when the relevance scores and response times of the IR servers are independent and identically distributed. When servers are heterogeneous, we present a simulations-based optimization technique and demonstrate how the optimal query set and wait time may be determined. The technique is computationally efficient and can be used to generate decision rules for source selection and query termination that are relatively easy to implement. We use data gathered from two different contexts - a DIR system that queries IR engines of several US federal agencies and a comparison shopping engine that queries multiple stores for price and product information - to validate our technique. Our research demonstrates that user satisfaction can be considerably improved by modeling user utility and incorporating historical information on performance of the IR servers.

Workshop on large-scale distributed systems for information retrieval

The Workshop on Large-Scale Distributed Systems for Information Retrieval was a venue for seminal ideas on the design of systems for search. The workshop focused mainly on mechanisms for P2P IR, which is currently a highly popular research area, but it also had fruitful discussions and presentations on other architectures for large-scale systems. Given the attendance and the good level of discussion, we conclude that systems for information retrieval is a growing and promising area of research.

Performance Analysis of Distributed Web Information Retrieval Systems

The importance and size of Web search engines is increasing daily. Information retrieval systems based on a single centralized index present several problems, which lead to the use of distributed information retrieval systems to effectively search for and locate the required information. In this study, we analyze two improvements over the brokers’ bottlenecks in a distributed information retrieval system. We demonstrate that reducing the partial results sets will improve the response time of a distributed system by 53%, with a negligible probability of changing the system’s precision and recall values. Finally, we present a simple hierarchical distributed broker model that will reduce the response times for a distributed system by 55%.

Result merging methods in distributed information retrieval with overlapping databases

In distributed information retrieval systems, document overlaps occur frequently among different component databases. This paper presents an experimental investigation and evaluation of a group of result merging methods including the shadow document method and the multi-evidence method in the environment of overlapping databases. We assume, with the exception of resultant document lists (either with rankings or scores), no extra information about retrieval servers and text databases is available, which is the usual case for many applications on the Internet and the Web. The experimental results show that the shadow document method and the multi-evidence method are the two best methods when overlap is high, while Round-robin is the best for low overlap. The experiments also show that [0,1] linear normalization is a better option than linear regression normalization for result merging in a heterogeneous environment.