Design of improved focused web crawler by analyzing semantic nature of URL and anchor text

Abstract

The world is completely working on digital data. The largest and prime or main collection of this digital data is web. The size of this web is increasing round-the-clock. The principal problem is to search this huge database for specific information. To state whether a web page is relevant to a search topic is a dilemma[l]. There are many techniques to state the relevancy but if focus on the users' perspective as key issue to guide search then semantic based web crawler are unsurpassed. Semantic based web crawlers maps relevancy with the help of lexical database. The crawler uses the senses provided by lexical database to discover relatedness among the search query and the web page being searched. Focused web crawler helps to find the similarity of web page to the search query without downloading that page. Thus focused web crawler is saving the bandwidth required to download a web page. This paper proposed and discuss one such approach to implement semantic based focused web crawler.