Deep Web Data Sources Research Articles

Towards an automated method to assess data portals in the deep web

The rising number of data portals has been increasing demand for new techniques to assess data openness in an automated manner. Some methods have emerged that presuppose well-organized data catalogs, the availability of API interfaces and natively exposed metadata. However, many data portals, particularly those of local governments, appear to be misimplemented and developed with the classic website model in mind, which provides access to data only through user interaction with web forms. Data in such portals resides in the hidden part of the web, as it is dynamically produced only in response to direct requests. This paper proposes an automated method for assessing government-related data in the deep web on the basis of compliance with open data principles and requirements. To validate our method, we apply it in an experiment using the government websites of the 27 Brazilian capitals. The method is fully carried out for 22 of the capitals' websites, resulting in the analysis of 5.6 million government web pages. The results indicate that the keyword search approach utilized in the method, along with the checking of web pages for multifield web forms, is effective for identifying deep web data sources, as 1.5% of web pages with potential government data that are analyzed are found to contain data stored in the deep web. This work contributes to the development of a novel method that allows for the continuous checking and identification of government data from surface web data portals. In addition, this method can be scaled and repeated to assure the widest possible content coverage.

Deep Web Data Source Classification Based on Text Feature Extension and Extraction

With the growth of volume of high quality information in the Deep Web, as the key to utilize this information, Deep Web data source classification becomes one topic with great research value. In this paper, we propose a Deep Web data source classification method based on text feature extension and extraction. Firstly, because the data source contains less text, some data sources even contain less than 10 words. In order to classify the data source based on the text content, the original text must be extended. In text feature extension stage, we use the N-gram model to select extension words. Secondly, we proposed a feature extraction and classification method based on Attention-based Bi-LSTM. By combining LSTM and Attention mechanism, we can obtain contextual semantic representation and focus on words that are closer to the theme of the text, so that more accurate text vector representation can be obtained. In order to evaluate the performance of our classification model, some experiments are executed on the UIUC TEL-8 dataset. The experimental result shows that Deep Web data source classification method based on text feature extension and extraction has certain promotion in performance than some existing methods.

Discovery and classification of deep web data sources

Discovery and classification of deep web data sources

Crawling ranked deep Web data sources

In the era of big data, the vast majority of the data are not from the surface Web, the Web that is interconnected by hyperlinks and indexed by most general purpose search engines. Instead, the trove of valuable data often reside in the deep Web, the Web that is hidden behind query interfaces. Since numerous applications, like data integration and vertical portals, require deep Web data, various crawling methods were developed for exhaustively harvesting a deep Web data source with the minimal (or near-minimal) cost. Most existing crawling methods assume that all the documents matched by queries are returned. In practice, data sources often return the top k matches. This makes exhaustive data harvesting difficult: highly ranked documents will be returned multiple times, while documents ranked low have small chance being returned. In this paper, we decompose this problem into two orthogonal sub-problems, i.e., query and ranking bias problems, and propose a document frequency based crawling method to overcome the ranking bias problem. The rational of our method is to use the queries whose document frequencies are within the specified range to avoid the effect of search ranking plus return limit and significantly reduce the difficulty of crawling ranked data source. The method is extensively tested on a variety of datasets and compared with two existing methods. The experimental result demonstrates that our method outperforms the two algorithms by 58 % and 90 % on average respectively.

Stratification-Based Outlier Detection over the Deep Web

For many applications, finding rare instances or outliers can be more interesting than finding common patterns. Existing work in outlier detection never considers the context of deep web. In this paper, we argue that, for many scenarios, it is more meaningful to detect outliers over deep web. In the context of deep web, users must submit queries through a query interface to retrieve corresponding data. Therefore, traditional data mining methods cannot be directly applied. The primary contribution of this paper is to develop a new data mining method for outlier detection over deep web. In our approach, the query space of a deep web data source is stratified based on a pilot sample. Neighborhood sampling and uncertainty sampling are developed in this paper with the goal of improving recall and precision based on stratification. Finally, a careful performance evaluation of our algorithm confirms that our approach can effectively detect outliers in deep web.

Survey of Techniques for Deep Web Source Selection and Surfacing the Hidden Web Content

Large and continuously growing dynamic web content has created new opportunities for large-scale data analysis in the recent years. There is huge amount of information that the traditional web crawlers cannot access, since they use link analysis technique by which only the surface web can be accessed. Traditional search engine crawlers require the web pages to be linked to other pages via hyperlinks causing large amount of web data to be hidden from the crawlers. Enormous data is available in deep web that can be useful to gain new insight for various domains, creating need to access the information from the deep web by developing efficient techniques. As the amount of Web content grows rapidly, the types of data sources are proliferating, which often provide heterogeneous data. So we need to select Deep Web Data sources that can be used by the integration systems. The paper discusses various techniques that can be used to surface the deep web information and techniques for Deep Web Source Selection.

Dynamic Non-Cooperative Structured Deep Web Selection

Most of structured deep web data sources are non-cooperation, therefore establish an accurate data source content summary by sampling is the core technology of data source selection. The content of deep web data source is updated from time to time, however existing efficient methods of non-cooperation structured data source selection does not consider summary update problem. Unrenewed summary of data source can not accurately characterize the content of the data source that produce a greater impact on data source selection. Base on this, we propose a dynamic data source selection method for non-cooperative structured deep web by combining subject headings sampling technology and subject headings extension technology. The experiment results show that our dynamic structured data source selection method has good recall ratio and precision besides being efficient.

Discover hidden web properties by random walk on bipartite graph

This paper proposes to use random walk (RW) to discover the properties of the deep web data sources that are hidden behind searchable interfaces. The properties, such as the average degree and population size of both documents and terms, are of interests to general public, and find their applications in business intelligence, data integration and deep web crawling. We show that simple RW can outperform the uniform random (UR) samples disregarding the high cost of UR sampling. We prove that in the idealized case when the degrees follow Zipf’s law, the sample size of UR sampling needs to grow in the order of O(N/ln 2 N) with the corpus size N, while the sample size of RW sampling grows logarithmically. Reuters corpus is used to demonstrate that the term degrees resemble power law distribution, thus RW is better than UR sampling. On the other hand, document degrees have lognormal distribution and exhibit a smaller variance, therefore UR sampling is slightly better.

Efficient Approach for Knowledge Management Using Deep Web Information Retrieval System

Along with the development of network technology, web information are rapidly growing, and the way of information storage is gradually changed from the html to the database, thus web information can be divided into the surface web and deep web. Deep web is a concept corresponding to the surface web. It means from ordinary search engine that is difficult to discover the information content of a web page. The traditional crawler only crawl the content on the surface of a web, which makes the current traditional search engine, did not retrieve deep web data. Deep web compared with surface web has the advantage of large volume, high quality, theme single-minded, good structured. In view of several advantages, the establishment of deep web data integration system is becoming a research hotspot. The deep web query interface is the only entrance of the background database, so how to determine which web form is the query interface is important to the deep web information access. However, because the page proportion on the internet which contains querying interface is very small, using the traditional breadth-first strategy and keyword filtering method to crawl, it will download a lot of unrelated pages, spend a lot of resources, we need a way to efficiently find and collect the query interfaces through deep web crawling strategy. We proposed novel query planning approach, for executing different types of complex attribute through queries over multiple inter-dependent deep web data sources. increase accelerate query searching based on attribute selection, execution and propose optimization techniques, including query plan merging and grouping optimization. Keywords: Novel query planning approach, Deep web, Semantic Deep Web, Ontologies, attribute.

Research on the Classification of Deep Web Data Sources Based on Weight Adjustment Technique

Research on the Classification of Deep Web Data Sources Based on Weight Adjustment Technique

Automatic discovery of Web Query Interfaces using machine learning techniques

The amount of information contained in databases available on the Web has grown explosively in the last years. This information, known as the Deep Web, is heterogeneous and dynamically generated by querying these back-end (relational) databases through Web Query Interfaces (WQIs) that are a special type of HTML forms. The problem of accessing to the information of Deep Web is a great challenge because the information existing usually is not indexed by general-purpose search engines. Therefore, it is necessary to create efficient mechanisms to access, extract and integrate information contained in the Deep Web. Since WQIs are the only means to access to the Deep Web, the automatic identification of WQIs plays an important role. It facilitates traditional search engines to increase the coverage and the access to interesting information not available on the indexable Web. The accurate identification of Deep Web data sources are key issues in the information retrieval process. In this paper we propose a new strategy for automatic discovery of WQIs. This novel proposal makes an adequate selection of HTML elements extracted from HTML forms, which are used in a set of heuristic rules that help to identify WQIs. The proposed strategy uses machine learning algorithms for classification of searchable (WQIs) and non-searchable (non-WQI) HTML forms using a prototypes selection algorithm that allows to remove irrelevant or redundant data in the training set. The internal content of Web Query Interfaces was analyzed with the objective of identifying only those HTML elements that are frequently appearing provide relevant information for the WQIs identification. For testing, we use three groups of datasets, two available at the UIUC repository and a new dataset that we created using a generic crawler supported by human experts that includes advanced and simple query interfaces. The experimental results show that the proposed strategy outperforms others previously reported works.

Stratified sampling for data mining on the deep web

In recent years, the deep web has become extremely popular. Like any other data source, data mining on the deep web can produce important insights or summaries of results. However, data mining on the deep web is challenging because the databases cannot be accessed directly, and therefore, data mining must be performed by sampling the datasets. The samples, in turn, can only be obtained by querying deep web databases with specific inputs. In this paper, we target two related data mining problems, association mining and differential rulemining. These are proposed to extract high-level summaries of the differences in data provided by different deep web data sources in the same domain. We develop stratified sampling methods to perform these mining tasks on a deep web source. Our contributions include a novel greedy stratification approach, which recursively processes the query space of a deep web data source, and considers both the estimation error and the sampling costs. We have also developed an optimized sample allocation method that integrates estimation error and sampling costs. Our experimental results show that our algorithms effectively and consistently reduce sampling costs, compared with a stratified sampling method that only considers estimation error. In addition, compared with simple random sampling, our algorithm has higher sampling accuracy and lower sampling costs.

Selecting queries from sample to crawl deep web data sources

This paper studies the problem of selecting queries to efficiently crawl a deep web data source using a set of sample documents. Crawling deep web is the process of collecting data from search interfaces by issuing queries. One of the major challenges in crawling deep web is the selection of the queries so that most of the data can be retrieved at a low cost. We propose to learn a set of queries from a sample of the data source. To verify that the queries selected from a sample also produce a good result for the entire data source, we carried out a set of experiments on large corpora including Gov2, newsgroups, wikipedia and Reuters. We show that our sampling-based method is effective by empirically proving that 1 The queries selected from samples can harvest most of the data in the original database; 2 The queries with low overlapping rate in samples will also result in a low overlapping rate in the original database; and 3 The size of the sample and the size of the terms from where to select the queries do not need to be very large. Compared with other query selection methods, our method obtains the queries by analyzing a small set of sample documents, instead of learning the next best query incrementally from all the documents matched with previous queries.

Quality Estimation of Deep Web Data Sources for Data Fusion

In a variety of domains, the amount of Web information grows rapidly, and the types of data sources are proliferating. Moreover, different data sources often provide heterogeneous or conflicting data, so we need to resolve data conflicts and find truth by data fusion. Currently, there are several advanced techniques that consider accuracy of sources, freshness of sources and dependencies between sources to solve the conflicts, and these strategies achieved good results. To improve the data fusion, we propose a quality estimation model of Deep Web data sources (DSQ). According to the characteristics of data fusion, our estimation model selects three dimensions of factors-data quality, interface quality and service quality-as estimation criteria, and estimates the quality of data sources. Then, we improve the data fusion using the estimation results. Experiment shows that our model can accurately estimate the quality of Deep Web data sources, and significantly improve the data fusion.

Domain-Relied QSOFM and its Application in Deep Web Topic Classification

To solve the problem of Deep Web data sources topic classification, this paper proposed a quantum self-organization feature mapping network model(DR-QSOFM)with a classification algorithm. DR-QSOFM combines quantum computation and traditional SOFM, and relies the feature vectors and target vectors incoordinately in different phases of training, making a more centralized distribution of winner neurons in competitive layer and more obvious boundaries among clusters. Some experiments are designed and done on the expanded TEL-8 dataset to test the validity of DR-QSOFM.

ETTA-IM: A deep web query interface matching approach based on evidence theory and task assignment

Integrating Deep Web data sources require highly accurate matches between the attributes of the query interfaces. While interface matching has received more attentions recently, current approaches are still not sufficiently perfect: (a) they all suppose that every interface attribute type has been predefined; (b) most of them combine multiple matchers taking into account different aspects of information about schema, but the weights of individual matchers are usually manually generated, and there may exist a high degree of inconsistency among different matchers; and (c) most of them only consider one-to-one matches of attributes over the interfaces and lack effective mathematical modeling. Therefore, a novel deep web query interface matching approach called ETTA-IM is proposed based on evidence theory and task assignment. Varied kinds of type recognizers are defined to identify the types of interface attributes which are used to divide the schema space into several schema subspaces. A modified D-S evidence theory is used to automatically combine multiple matchers and to solve high conflicts among different matchers. One-to-one match decision is converted to extended task assignment problem and some tree structure heuristic rules are used to perform one-to-many match decision. Experiments show that ETTA-IM approach yields high precision and recall measures.

Ranking bias in deep web size estimation using capture recapture method

Many deep web data sources are ranked data sources, i.e., they rank the matched documents and return at most the top k number of results even though there are more than k documents matching the query. While estimating the size of such ranked deep web data source, it is well known that there is a ranking bias—the traditional methods tend to underestimate the size when queries overflow (match more documents than the return limit). Numerous estimation methods have been proposed to overcome the ranking bias, such as by avoiding overflowing queries during the sampling process, or by adjusting the initial estimation using a fixed function. We observe that the overflow rate has a direct impact on the accuracy of the estimation. Under certain conditions, the actual size is close to the estimation obtained by unranked model multiplied by the overflow rate. Based on this result, this paper proposes a method that allows overflowing queries in the sampling process.

Estimating deep web data source size by capture–recapture method

This paper addresses the problem of estimating the size of a deep web data source that is accessible by queries only. Since most deep web data sources are non-cooperative, a data source size can on...