Deep Web Pages Research Articles

An Efficient Mechanism for Deep Web Data Extraction Based on Tree-Structured Web Pattern Matching

The World Wide Web comprises of huge web databases where the data are searched using web query interface. Generally, the World Wide Web maintains a set of databases to store several data records. The distinct data records are extracted by the web query interface as per the user requests. The information maintained in the web database is hidden and retrieves deep web content even in dynamic script pages. In recent days, a web page offers a huge amount of structured data and is in need of various web-related latest applications. The challenge lies in extracting complicated structured data from deep web pages. Deep web contents are generally accessed by the web queries, but extracting the structured data from the web database is a complex problem. Moreover, making use of such retrieved information in combined structures needs significant efforts. No further techniques are established to address the complexity in data extraction of deep web data from various web pages. Despite the fact that several ways for deep web data extraction are offered, very few research address template-related issues at the page level. For effective web data extraction with a large number of online pages, a unique representation of page generation using tree-based pattern matches (TBPM) is proposed. The performance of the proposed technique TBPM is compared to that of existing techniques in terms of relativity, precision, recall, and time consumption. The performance metrics such as high relativity is about 17-26% are achieved when compared to FiVaTech approach.

Crawling the Deep Web Using Asynchronous Advantage Actor Critic Technique

In the digital world, World Wide Web magnitude is expanding very promptly. Now a day, a rising number of data-centric websites require a mechanism to crawl the information. The information accessible through hyperlinks can easily be retrieved with general-purpose search engines. A massive chunk of the structured information is invisible behind the search forms. Such immense information is recognized as the deep web and has structured information as compared to the surface web. Crawling the content of deep web is very challenging and requires filling the search forms with suitable queries. This paper proposes an innovative technique using an Asynchronous Advantage Actor-Critic (A3C) to explore the unidentified deep web pages. It is based on the policy gradient deep reinforcement learning technique that parameterizes the policy and value function based on the reward system. A3C has one coordinator and various agents. These agents learn from different environments, update the local gradients to a coordinator, and produce a more stable system. The proposed technique has been validated with Open Directory Project (ODP). The experimental outcome shows that the proposed technique outperforms most of the prevailing techniques based on various metrics such as average precision-recall, average harvest rate, and coverage ratio.

Deep Web crawling: a survey

Deep Web crawling refers to the problem of traversing the collection of pages in a deep Web site, which are dynamically generated in response to a particular query that is submitted using a search form. To achieve this, crawlers need to be endowed with some features that go beyond merely following links, such as the ability to automatically discover search forms that are entry points to the deep Web, fill in such forms, and follow certain paths to reach the deep Web pages with relevant information. Current surveys that analyse the state of the art in deep Web crawling do not provide a framework that allows comparing the most up-to-date proposals regarding all the different aspects involved in the deep Web crawling process. In this article, we propose a framework that analyses the main features of existing deep Web crawling-related techniques, including the most recent proposals, and provides an overall picture regarding deep Web crawling, including novel features that to the present day had not been analysed by previous surveys. Our main conclusion is that crawler evaluation is an immature research area due to the lack of a standard set of performance measures, or a benchmark or publicly available dataset to evaluate the crawlers. In addition, we conclude that the future work in this area should be focused on devising crawlers to deal with ever-evolving Web technologies and improving the crawling efficiency and scalability, in order to create effective crawlers that can operate in real-world contexts.

Design and implementation of crawling algorithm to collect deep web information for web archiving

PurposeThe purpose of this paper is to describe the development of an algorithm for realizing web crawlers that automatically collect dynamically generated webpages from the deep web.Design/methodology/approachThis study proposes and develops an algorithm to collect web information as if the web crawler gathers static webpages by managing script commands as links. The proposed web crawler actually experiments with the algorithm by collecting deep webpages.FindingsAmong the findings of this study is that if the actual crawling process provides search results as script pages, the outcome only collects the first page. However, the proposed algorithm can collect deep webpages in this case.Research limitations/implicationsTo use a script as a link, a human must first analyze the web document. This study uses the web browser object provided by Microsoft Visual Studio as a script launcher, so it cannot collect deep webpages if the web browser object cannot launch the script, or if the web document contains script errors.Practical implicationsThe research results show deep webs are estimated to have 450 to 550 times more information than surface webpages, and it is difficult to collect web documents. However, this algorithm helps to enable deep web collection through script runs.Originality/valueThis study presents a new method to be utilized with script links instead of adopting previous keywords. The proposed algorithm is available as an ordinary URL. From the conducted experiment, analysis of scripts on individual websites is needed to employ them as links.

Deep Web Exploration with Learning-Based Application Framework

The inaccessible nature of the Deep Web motivated us to propose a learning-based application framework for exploring Deep Web, based on the query provided by the user, through TOR interface. The proposed framework, DWIRS: Deep Web Indexing and Ranking Search, employs two algorithms: BSVM (Boosted Support Vector Machine) for indexing, and, BRF (Boosted Random Forest) for ranking Deep Web pages. To evaluate the performance of DWIRS framework, DWIRS Document Clustering Workbench, a standalone GUI application has been developed and validated. A mobile application for exploring Deep Web has also been developed. The purposed DWIRS framework is efficient.

A Survey on Uniform Resource Locator and Content Matching to Discover Deep- Web Pages

Objectives: 1) The Objective is to harvest the deep web pages efficiently1 2) Personalize search according to user interest. 3) Combine pre-query and post-query approach. Methods/Statistical Analysis: Three methods are used 1. URL Matching: This method is used to match the query content in URL. For that, the system gets a link from online and site database. Links are extracted to match the user entered query content. 2. Content matching: This method is used extracting the links and getting form content and matching the user-entered query .If match calculates the occurrence frequency of that query on the form. For that, it use Jsoup library.3. Pre-query Algorithm: This method used to display pre-query result after entering focus in the search box. For that user login to the system that time system select his profile and according to that links will display to the user. Findings: As in the existing system, most of the search engines display the results according to the most visited sites or recently added sites. To find the deep web pages from the databases is a challenging, because they are not enrolled with any web indexes and keep constantly changing. In this system, Smart Crawler performs URL matching and content matching to discover the deep web pages. Proposed crawler proficiently gets deep-web network from wide destination and accomplishes the higher outcome from different crawlers. Page ranking is performed and it displays high ranked results on the result page .Here it provided personalized search, results display according to the user professions. Maintaining log file and the pre-query result will reduce time. First time this crawler perform personalize search this means that this crawler is unique. During the evaluation, notice that proposed approach is more efficient than the existing crawler. Application/Improvements: The application gathers real-time user profile information from user accounts. Therefore, it must be reliable and keep those data in safe. This crawler is used as the search engine for e-learning application, E-shopping site. Links can be a bookmark for future use. As in improvement, it can rank the pages according to user-entered review for each link. Also, the opened link will display page content in task extraction form. That is code, concept, URL on the page. Keyword: Deep Web, IP, Positioning, Smart Crawler

Robots exclusion and guidance protocol

With the rapid development of the Internet, general-purpose web crawlers have increasingly become unable to meet people's individual needs as they are no longer efficient enough to fetch deep web pages. The presence of several deep web pages in the websites and the widespread use of Ajax make it difficult for general-purpose web crawlers to fetch information quickly and efficiently. On the basis of the original Robots Exclusion Protocol (REP), a Robots Exclusion and Guidance Protocol (REGP) is proposed in this paper, by integrating the independent scattered expansions of the original Robots Protocol developed by major search engine companies. Our protocol expands the file format and command set of the REP as well as two labels of the Sitemap Protocol. Through our protocol, websites can express their aspects of requirements for restrictions and guidance to the visiting crawlers, and provide a general-purpose fast access of deep web pages and Ajax pages for the crawlers, and facilitates crawlers to easily obtain the open data on websites effectively with ease. Finally, this paper presents a specific application scenario, in which both a website and a crawler work with support from our protocol. A series of experiments are also conducted to demonstrate the efficiency of the proposed protocol.

Identification of Query Forms for Retrieving the Information From Deep Web

Web databases are now present everywhere. The data from the Deep Web can not be accessed by Search engine and web crawlers directly. The only way to access the hidden database is through query interfaces and filling up number of HTML forms for a specific domain [18]. In this paper a technique called QFORT (QUERY FORM RETRIEVAL TECHNIQUE) has been developed for identifying the relevant query form is presented. Retrieving information from deep web pages using wrappers is a fundamental problem arising in a huge range of web pages of vast practical interests. In this paper, we propose a novel technique to the problem of identifying the query forms from Web pages, which is one of the key problems in automatic extraction approach. The problem is resolved by many authors by using different technique Intensive experiments on real web sites show that the proposed technique can effectively help extracting desired data with high accuracies in most of the cases.

Automatic Deep Web Query Results User Satisfaction Evaluation with Click-through Data Analysis

We browse through hundreds of Deep web pages everyday to find information of interest. We feel happy when Deep web browsing operations provide us with necessary information; otherwise, we feel bitter. Now, the measurement of this user satisfaction has become a hot research topic. In this paper, we propose a click-through-data-based and unsupervised user satisfaction evaluation system, CNEITE, to evaluate the user satisfaction of Deep Web query result pages. It applies query type classifying, navigational query evaluating, informational/transactional query evaluating to solve the challenging tasks. We evaluated our CNEITE system on the AOL data sets, experimental results show that CNEITE achieves higher classify precision than a widely used classify method , Dtree, and higher annotate answer accuracy than method proposed in [17].

Annotation for Query Result Records Based on Domain-Specific Ontology

The World Wide Web is enriched with a large collection of data, scattered in deep web databases and web pages in unstructured or semi structured formats. Recently evolving customer friendly web applications need special data extraction mechanisms to draw out the required data from these deep web, according to the end user query and populate to the output page dynamically at the fastest rate. In existing research areas web data extraction methods are based on the supervised learning (wrapper induction) methods. In the past few years researchers depicted on the automatic web data extraction methods based on similarity measures. Among automatic data extraction methods our existing Combining Tag and Value similarity method, lags to identify an attribute in the query result table. A novel approach for data extracting and label assignment called Annotation for Query Result Records based on domain specific ontology. First, an ontology domain is to be constructed using information from query interface and query result pages obtained from the web. Next, using this domain ontology, a meaning label is assigned automatically to each column of the extracted query result records.

Data Extraction and Annotation for Web Databases using Multiple Annotators Approach - A Review

contain huge amount of information on Web sites the user can retrieve this with help of the search input query to Web databases & fetch the relevant information. Perhaps Web databases return the multiple search output records dynamically on Web browser, these search record are containing the Deep Web pages in the form of HTML pages. It is time consuming &human efforts are involved. The traditional search engine does not index the hidden Web pages from Web databases, such as (Google, Yahoo etc.). Many existing proposed techniques have addressed the problem of how to extract efficient structure data from Deep Web. The deep web refers to the hidden database used by web sites. But the information extraction & annotation is key challenge in web mining. The information retrieval should be done automatically & arrange in a systematic way for further processing. Various methodologies like wrapper induction is been induced. The labeling is done to the extracted information as per the concept.Various types of annotators are used on the basis of the data to be annotated. In this paper survey the automatic annotation approach on the basis of different feature of text node and data units.

DWDE-IR: An Efficient Deep Web Data Extraction for Information Retrieval on Web Mining

Deep Web is a widely unexplored data source, is becoming an important research topic. Retrieving structured data from deep web pages is the challenging problem due to their complex structure. In this paper, Information extracts on the Deep Web pages based on the Deep Web Data Extraction technique (DWDR-IR). Search engines usually return a large number of pages in response to the user queries. To help the users to navigate in the result list, ranking methods are activated on the search results. In this paper, a page ranking mechanism called Coherence Ratio based Page (CRP) ranking algorithm is used. To retrieve the information accurately, an approach called WordNet is used. WordNet checks the similarity of data records and find the correct data region with higher precision using the semantic properties of data records. This concept is very important to display the valuable results occur on the top of the result list on the basis of browsing behavior of the user, it reduces the search space and provides high accuracy. This approach handles the visual features on the deep web data extraction, including data item extraction, data record extraction and visual wrapper generation. The proposed work removes all noise such as header, footer, irrelevant advertisement and irrelevant content using NoiSe Filter (NSFilter) algorithm. The proposed method retrieves perfect extraction of relevant results from the deep web pages. DWDE-IR results higher precision, recall and filter accuracy than the existing method ViDE.

I-ViDE: An Improved Vision-Based Approach for Deep Web Data Extraction

Deep Web contents are accessed by queries submitted to Web databases and the returned data records are enwrapped in dynamically generated Web pages (they will be called deep Web pages in this paper). Extracting structured data from deep Web pages is a challenging problem due to the underlying intricate structures of such pages. Until now, a large number of techniques have been proposed to address this problem, but all of them have inherent limitations because they are HTML language dependent .Visual features are not taken into consideration. All previous methods are mostly dependent on table tags. A Vision based approach for web data extraction has overcome the limitations of previous work by utilizing some interesting common visual features on the web page. But still this approach has one drawback that it can process web page containing only one data region. Due to processing of one data region it reduces the precision and recall rate. As precision give us the rate that how many correct data records are extracted from relevant data records and recall give us the rate that how many relevant data records are extracted from overall data records. The proposed ImprovedViDE approach handles multi data-region in deep web pages which can improve the precision rate and recall rate.

Manufacturing Deep Web Service Management: Exploring Semantic Web Technologies

Since the wide adoption of database and Web service technologies in cross-enterprise manufacturing collaboration, leveraging deep Web contents through service discovery has become one of the most advanced trends in the manufacturing field. However, the effective service discovery and knowledge retrieval from manufacturing deep Web services is still a critical issue because of the underlying intricate structures of deep Web pages. This motivates us to seek a different way to manage manufacturing deep Web services by semantically annotating them to reflect their hidden, dynamic, and heterogeneous contents. In this article, a novel ontology-based approach that uses semantic Web paradigm is proposed, and its implementation in real applications is validated.

Visual Architecture based Web Information Extraction

The World Wide Web has more online web database which can be searched through their web query interface. Deep Web contents are accessed by queries submitted to Web databases and the returned data records are enwrapped in dynamically generated Web pages. Extracting structured data from deep Web pages is a challenging task due to the underlying complicate structures of such pages. Until now, a large number of techniques have been proposed to address this problem, but all of them have inherent limitations because they are Web-page- programming-language dependent. As the popular two- dimensional media, the contents on Web pages are always displayed regularly for users to browse. This motivates us to seek a different way for deep Web data extraction to overcome the limitations of previous works by utilizing some interesting common visual features on the deep Web pages. In this paper, a novel vision-based approach that is Web- page programming- language-independent is proposed. This approach primarily utilizes the visual features on the deep Web pages to implement deep Web data extraction, including data record extraction and data item extraction.

ViDE: A Vision-Based Approach for Deep Web Data Extraction

Deep Web contents are accessed by queries submitted to Web databases and the returned data records are enwrapped in dynamically generated Web pages (they will be called deep Web pages in this paper). Extracting structured data from deep Web pages is a challenging problem due to the underlying intricate structures of such pages. Until now, a large number of techniques have been proposed to address this problem, but all of them have inherent limitations because they are Web-page-programming-language-dependent. As the popular two-dimensional media, the contents on Web pages are always displayed regularly for users to browse. This motivates us to seek a different way for deep Web data extraction to overcome the limitations of previous works by utilizing some interesting common visual features on the deep Web pages. In this paper, a novel vision-based approach that is Web-page-programming-language-independent is proposed. This approach primarily utilizes the visual features on the deep Web pages to implement deep Web data extraction, including data record extraction and data item extraction. We also propose a new evaluation measure revision to capture the amount of human effort needed to produce perfect extraction. Our experiments on a large set of Web databases show that the proposed vision-based approach is highly effective for deep Web data extraction.

QA-Pagelet: data preparation techniques for large-scale data analysis of the deep Web

This paper presents the QA-Pagelet as a fundamental data preparation technique for large-scale data analysis of the deep Web. To support QA-Pagelet extraction, we present the Thor framework for sampling, locating, and partioning the QA-Pagelets from the deep Web. Two unique features of the Thor framework are 1) the novel page clustering for grouping pages from a deep Web source into distinct clusters of control-flow dependent pages and 2) the novel subtree filtering algorithm that exploits the structural and content similarity at subtree level to identify the QA-Pagelets within highly ranked page clusters. We evaluate the effectiveness of the Thor framework through experiments using both simulation and real data sets. We show that Thor performs well over millions of deep Web pages and over a wide range of sources, including e-commerce sites, general and specialized search engines, corporate Web sites, medical and legal resources, and several others. Our experiments also show that the proposed page clustering algorithm achieves low-entropy clusters, and the subtree filtering algorithm identifies QA-Pagelets with excellent precision and recall.