Distributed Search Engine Research Articles

Research on an Application-layer Network Performance Measure Method

This paper propose an application-layer network performance measure method for distributed information collection system, and verified the authenticity and stability of the proposed network distance by experiments. Based on the research process, we proposed a definition of network distance based on distributed information collection system of distributed search engine, and to reduce the network of distributed data acquisition system from the total amount of the research for this article, is the main direction. Aimed at the definition of the network distance in information collection system, improved network distance prediction algorithm and Web partition system, experiments and analysis are all carried out. We finish an experimental verification of application-layer network distance that limit the authenticity and stability in statistics less than 10%, through which as a standard of network distance of establishing network coordinate, also can provide a theoretical basis for the further network distance prediction.

An Approach to XML Information Retrieval in Distributed Systems

Abstract Modern search engines are quite successful in finding relevant documents by using Information Retrieval techniques. However, the centralized architecture of systems such as Google raises concerns in regard of the possibility to censure and to manipulate information. Promising alternatives are peer-to-peer search engines that organize themselves without any central index. We propose a distributed search engine that exploits XML structure of documents in order to improve effectiveness (search quality) and efficiency (resources) of the search. The developed system is based on a set of equal and autonomous peers. It is the first solution of its kind worldwide for distributed XML-Retrieval.

Retrieval of Fresh Information Using P2P Network

The fast development of the World Wide Web and Dynamic nature makes it a challenge for searching and retrieving of information that is more recent .The WWW is a rapidly growing and changing information source. Its growth and change rates make the task of finding recent information harder. The exponentially growing information published on the Web relies largely on a few major search engines like Google to be brought to the public nowadays. In fact, the increasing dynamics of the information distributed on the Internet challenge the flexibility of these centralized search engines. However, such an engine is not suitable for fresh information retrieval because it spends a long time to collect web pages by using a web robot (or crawler). In the other hand, there are some distributed search engines such as Cooperative Search Engine (CSE), and so on. However, these distributed search engines are able to update in a very short time, e.g. a few minutes. But it has more communication delay. In this paper, in order to reduce the delay we proposed a P2P Web search that connects an a-priori unlimited number of peers, each of which maintains a personal local database and a local search facility. Each peer posts a small amount of metadata to a physically distributed directory layered on top of a DHT-based overlay network that is used to efficiently select promising peers from across the peer population that can best locally execute a query. Further usage of Apriori algorithm and Page Rank algorithm improves mining of frequently viewed documents to greater extent. We also uses Textual Entailment approach for searching a relevant document for a given keyword ie) to retrieve not only textual documents that have specified keywords, but also to discover semantically equivalent or entailed documents from given keywords.

Cooperative cache analysis for distributed search engines

In this paper, we study the performance of a distributed search engine from a data caching point of view using statistical tools on a varied set of configurations. We study two strategies to achieve better performance: cache-aware load balancing that issues the queries to nodes that store the computation in cache; and cooperative caching (CC) that stores and transfers the available computed contents from one node in the network to others. Since cache-aware decisions depend on information about the recent history, we also analyse how the ageing of this information impacts the system performance. Our results show that the combination of both strategies yield better throughput than individually implementing cooperative cache or cache-aware load balancing strategies because of a synergic improvement of the hit rate. Furthermore, the analysis concludes that the data structures to monitor the system need only moderate precision to achieve optimal throughput.

Redundant architecture in Cooperative Search Engine

We have developed a distributed search engine, called Cooperative Search Engine (CSE), to retrieve fresh information. In CSE, a local search engine located in each web server makes an index of local pages, and a meta search server integrates these local search engines to realise a global search engine. This meta server is a single point of failure in CSE. Therefore, we propose the introduction of redundancy for the meta search servers to increase the availability of CSE. In this paper, we describe and evaluate a reliable architecture for CSE.

Design of distributed search engine system for apparel based on semantic Web services

According to the needs of supply-chain management in the environment of E-business,the authors analyzed the problems existing in the present search engine for apparel-goods and proposed a new distributed search engine for apparel goods based on semantic Web Services(WS),and then discussed its architecture.After introducing the apparel Ontology design model and the representation method based on Ontology Web Language(OWL),the basic functions of apparel search engine and WS synthesized in distributed environment were presented.Theoretical analysis and the designed prototype show that this search engine,based on apparel semantic tree,is much more efficient than full-text search engine when searching with multi-

P2pDating: Real life inspired semantic overlay networks for Web search

We consider a network of autonomous peers forming a logically global but physically distributed search engine, where every peer has its own local collection generated by independently crawling the Web. A challenging task in such systems is to efficiently route user queries to peers that can deliver high quality results and be able to rank these returned results, thus satisfying the users’ information need. However, the problem inherent with this scenario is selecting a few promising peers out of an a priori unlimited number of peers. In recent research a rather strict notion of semantic overlay networks has been established. In most approaches, peers are connected to other peers based on a rigid semantic profile by clustering them based on their contents. In contrast, our strategy follows the spirit of peer autonomy and creates semantic overlay networks based on the notion of “peer-to-peer dating”. Peers are free to decide which connections they create and which they want to avoid based on various usefulness estimators. The proposed techniques can be easily integrated into existing systems as they require only small additional bandwidth consumption as most messages can be piggybacked onto established communication. We show how we can greatly benefit from these additional semantic relations during query routing in search engines, such as Minerva, and in the JXP algorithm, which computes the PageRank authority measure in a completely decentralized manner.

Software Agent Mobility

This issue is the first of a two issue collection of selected papers from the annual AIMS (Agents, Interactions, Mobility, and Systems) conference track spanning 2002-2006. AIMS began in 2002 as part of the ACM SAC (Symposium on Applied Computing) and continued for five years. The first conference was held in Madrid, Spain. Subsequent conferences were held in Melbourne (Florida, USA), Nicosia, Cyprus, Santa Fe, New Mexico, and Dijon, France. The track was primarily created to provide a venue for applied topics in software agents but became the primary venue for papers on mobile agents, as the IEEE conference on Mobile Agents was discontinued beyond 2002. Hence, the initial collection of six papers from the AIMS track focuses exclusively on topics related to mobile agents. In the first paper, Binder and Roth examine the security capabilities of the Java platform with respect to supporting mobile agents. The paper raises the question as to whether Java is an appropriate language and has the necessary security mechanisms to safely support mobile agents. After highlighting the missing capabilities, the paper concludes by pointing to ongoing work that fortify shortcomings in Java. The second paper, by Bettini, Nicola, and Loreti, describes their X-KLAIM programming language that is designed to simplify the task of writing mobile agents as well as distributed applications in general. X-KLAIM builds on the concept of tuple spaces as introduced by the Linda programming language. The paper presents examples of distributed applications and mobile agents that are written in the X-CLAIM language. The third paper, by Quitadamo, Leonardi, and Cabri, describe an approach to provide strong mobility for agents. Strong mobility implies that the execution state of the treads of the agent are preserved, thereby allowing the agent to continue execution after reaching the destination node. Very few mobile agent systems are capable of supporting strong migration. Their approach takes advantage of a modified Jikes RVM (Research VM) from IBM and supports mobile agents written in the Aglets programming language. In the fourth paper, Moreau addresses the problem of routing messages to agents that are roaming a network. His approach begins with a fault-tolerant directory service that is used to keep track of agent locations and messages being sent. Nodes use forwarding pointers when agents move and the system provides redundancy in order to handle failure of nodes. The fifth paper, by Overeinder, de Groot, Wijngaards, and Brazier addresses the problem of agent migration across heterogeneous platforms. Their approach aims to provide cross-platform compatibility via authoring the mobile agent in a meta-language that provides a blueprint for the agent's behavior. Each platform then contains an agent factory that generates the code for the agent based on the blueprint. Finally, in the sixth paper, Roth, Peters, and Pinsdorf describe an application of mobile agents to search for multimedia by content-matching. Instead of transmitting large multimedia objects to clients, a distributed search engine is used that embeds the features of interest into a mobile agent that is dispatched to the sources of the images. Content matching is then performed on the provider nodes, thereby reducing the need to unnecessarily transmit large amounts of data over the network. Finally, we would like to express our gratitude to everyone who worked with us in the Program Committee of the AIMS Workshop. THANK YOU! Henry Hexmoor, Marcin Paprzycki, Niranjan Suri.

DAME: Searching Large Data Sets Within a Grid-Enabled Engineering Application

The use of search engines within the Internet is now ubiquitous. This work examines how Grid technology may affect the implementation of search engines by focusing on the Signal Data Explorer application developed within the Distributed Aircraft Maintenance Environment (DAME) project. This application utilizes advanced neural-network-based methods (Advanced Uncertain Reasoning Architecture (AURA) technology) to search for matching patterns in time-series vibration data originating from Rolls-Royce aeroengines (jet engines). The large volume of data associated with the problem required the development of a distributed search engine, where data is held at a number of geographically disparate locations. This work gives a brief overview of the DAME project, the pattern marching problem, and the architecture. It also describes the Signal Data Explorer application and provides an overview of the underlying search engine technology and its use in the aeroengine health-monitoring domain.

A Distributed Content-Based Search Engine Based on Mobile Code and Web Service Technology

Current search engines crawl the Web, download content, and digest this content locally. For multimedia content, this involves considerable volumes of data. Furthermore, this process covers only publicly available content because content providers are concerned that they otherwise loose control over the distribution of their intellectual property. We present the prototype of our secure and distributed search engine, which dynamically pushes content based feature extraction to image providers. Thereby, the volume of data that is transported over the network is significantly reduced, and the concerns mentioned above are alleviated. The distribution of feature extraction and matching algorithms is done by mobile software agents. Subsequent search requests performed upon the resulting feature indices by means of remote feature comparison can either be realized through mobile software agents, or by the use of implicitly created Web services which wrap the remote comparison functionality, and thereby improve the interoperability of the search engine. We give a description of the search engine's architecture and implementation, depict our concepts to integrate agent and Web service technology, and present quantitative evaluation results. Furthermore, we discuss related security mechanisms for content protection and server security.

A topic-specific Web robot model based on restless bandits

Constructing and maintaining topic-specific Web indexes is modeled by a restless-bandits generalization and resolved by a reinforcement-learning algorithm. The authors outline the potential role of topic-specific robots in distributed search engine design, and they model the complex problem of automatically constructing and maintaining topic-specific Web indexes. Experimental results establish the viability of a topic-specific Web robot design based on the restless bandit model. The results indicate that the proposed algorithm is a good foundation on which to build a complete solution.

The metaknowledge-based intelligent routing system (MIRS)

This paper addresses the issue of locating relevant information in a network of heterogeneous, unfederated information bases of various types, including structured databases, text, audio, picture and video files. The problem is to determine where the required information resides in a network, in locations unknown to the user. The objective is to construct a user-friendly, intelligent, search and routing mechanism in order to find the most relevant information bases in the network. We introduce a mechanism for presenting queries, routing queries, updating knowledge, and learning in a metaknowledge base (MKB). This has been named the metaknowledge-based intelligent routing system (MIRS). MIRS finds the location of the desired information by its ability to “understand” the user’s query and to access information by content, rather than by address. MIRS behaves like a distributed search engine, working with a distributed metaknowledge index-file. There is no need for periodic web-crawling, web-robots, or agents of any sort. The network itself encapsulates the knowledge and routing algorithms that provide the user access-by-content to the relevant information. Contrary to web servers, the specific MIRS servers are not linked by hypertext links, but rather by knowledge links, randomly acquired or expertly built. The system also differs from the usual search engines in that it is capable of handling different types of media (e.g., text, database, multimedia) and applies natural language parsing techniques to understand the intention of the user, as well as potentially use a user-profile to enhance the original query before distributing it over the network. The “metadata” describing the information bases are spread across a network of routing and information servers and are modified as a result of search operations and introduction of new information bases into the system.

A Full-Text Search System Covering the Whole GenomeNet

Retrieval of entries by keywords is one of the most traditional services for genome databases. DBGET [1] service in GenomeNet [2] is a typical one which provides keyword search and entry retrieval against about 20 heterogeneous databases (GenBank, EMBL, SWISS-PROT, PDB, KEGG, etc.). In general, such a service is based on keyword indices to entries, which is generated from restricted fields of entry headers, that is, entry identifiers, names of genes and proteins, and so on. Besides this type of approach has advantages in retrieval with high precision, its recall remains low since all the keywords which only occur in the ignored fields do not hit. Furthermore, specialized indexing process, which is sensitive to the description style of fields, tends to need long time computation for generating indices. On the other hand, as WWW grows and grows, technology of full-text search is drastically progressing in these years. Since GenomeNet contains approximately 10 millions of entries, it might be promising to use freely distributed search engines for indexing and keyword search against the whole GenomeNet in practical and sufficient response time. In this poster, a result of such application study is reported.

ProFusion: Intelligent Fusion from Multiple, Distributed Search Engines

ProFusion: Intelligent Fusion from Multiple, Distributed Search Engines