Hidden Web Pages Research Articles

Crawling the Hidden Web: An Approach to Dynamic Web Indexing

majority of the websites encapsulating online information are dynamic and hence too sophisticated for many traditional search engines to index. With the ever growing quantity of such hidden web pages, this issue continues to raise diverse opinions between the research and practitioner among the web mining communities. Several aspects enriching these dynamic web pages are bringing more challenges day-by-day to index them. By explaining these aspects and challenges, in this paper we have presented a framework for dynamic web indexing. With the implementation of this framework and the results which we have found from it, all the necessary experimental setup and the developmental processes are explained. We have concluded by exposing a possible future scope through the integration of Hadoop-Mapreduce with this framework to update and maintain the index.

Crawling the Deep Web: A Study

Search interfaces are only the mode to retrieve the information on the web, which is becoming enormous day by day. One has to key in the set of keywords in search form to retrieve pages from the website. These hidden web pages do not have static links[1], hence, search engines are unable to get and index such pages and thus no result is returned. As per recent studies many of the deep webs are of tremendous high quality, which can be extremely useful to the users. These pages are often referred to as the Hidden Web or the Deep Web. A powerful web crawler can discover and download hidden web pages. The main challenge that a hidden web crawler has to face is to make meaningful queries to issue to the site as the only entry point to the hidden website is query interface. Here a theoretical framework to investigate the query generation problem is provided for the hidden web and effective policies for generating queries automatically studies.