Information Retrieval in Web Crawling Using Population Based, and Local Search Based Meta-heuristics: A Review

Abstract

The exponential growth and dynamic nature of the world wide web has created challenges for the traditional Information Retrieval (IR) methods. Both issues are the imperative source of problems for locating the information on web. The crawlers expedite web based information retrieval systems by following hyperlinks in web pages to automatically download new and updated content. The web crawlers systematically traverse the web pages, and fetch the information viz. nature of the web content, hyper-links present on the web page, etc. This paper reviews and compares the meta-heuristic approaches like population based, evolutionary algorithms, and local search used for IR in web crawling. This paper reviews how these techniques has been developed, enhanced and applied.