Optimal allocation of heterogeneous robots in world wide web search engines

Abstract

It is a conflicting requirement to reduce network traffic and loads on Web servers caused by the robots in a World Wide Web search engine and to increase the currency of the database in the search engine. This paper addresses the problem of finding optimal allocations of a given number of robots in a search engine to heterogeneous Internet domains such that a cost function which characterizes the combined consideration of network/server loads and database currency can be minimized. The cost function and the optimization problem are formulated based on a queueing model of search engines. Our algorithm to solve the problem uses the depth-first search strategy to traverse an enumeration tree which generates all possible allocations.