Effective page refresh policy

Abstract

AbstractWeb pages are created or updated randomly. As for a search engine, keeping up with the evolving Web is necessary. But previous studies have shown the crawler's refresh ability is limited because it is not easy to detect the change instantly, especially when the resources are limited. This article concerns modeling on an effective Web page refresh policy and finding the refresh interval with minimum total waiting time. The major concern is how to model the change and which part should be updated more often. Toward this goal, the Poisson process is used to model the process. Further, the relevance is also used to adjust the process, and the probability on some sites is higher than others so these sites will be given more opportunities to be updated. It is essential when the bandwidth is not wide enough or the resource is limited. The experimental results validate the feasibility of the approach. On the basis of the above works, an educational search engine has been developed. © 2007 Wiley Periodicals, Inc. Comput Appl Eng Educ 14: 240–247, 2007; Published online in Wiley InterScience (www.interscience.wiley.com); DOI 10.1002/cae.20155