Web Data Integration Research Articles

Deep Web Data Extraction Based on Regular Expression

Data extraction is an important issue in Deep web data integration. In order to extract the query results of the Deep Web, it is firstly required to locate the target data block correctly. Due to the html source code of web pages can be parsed as well structured DOM, we proposed an effective algorithm for discerning the common path based on hierarchical DOM. Based on the common path and our predefined regular expression, the target data of the Deep Web can be extracted effectively. The experimental results on real websites show that our proposed algorithm is highly effective.

Active XML-based Web data integration

Today, the Web is the largest source of information worldwide. There is currently a strong trend for decision-making applications such as Data Warehousing (DW) and Business Intelligence (BI) to move onto the Web, especially in the cloud. Integrating data into DW/BI applications is a critical and time-consuming task. To make better decisions in DW/BI applications, next generation data integration poses new requirements to data integration systems, over those posed by traditional data integration. In this paper, we propose a generic, metadata-based, service-oriented, and event-driven approach for integrating Web data timely and autonomously. Beside handling data heterogeneity, distribution and interoperability, our approach satisfies near real-time requirements and realize active data integration. For this sake, we design and develop a framework that utilizes Web standards (e.g., XML and Web services) for tackling data heterogeneity, distribution and interoperability issues. Moreover, our framework utilizes Active XML (AXML) to warehouse passive data as well as services to integrate active and dynamic data on-the-fly. AXML embedded services and changes detection services ensure near real-time data integration. Furthermore, the idea of integrating Web data actively and autonomously revolves around mining events logged by the data integration environment. Therefore, we propose an incremental XML-based algorithm for mining association rules from logged events. Then, we define active rules dynamically upon mined data to automate and reactivate integration tasks. Finally, as a proof of concept, we implement a framework prototype as a Web application using open-source tools.

Multi-objective optimization integration of query interfaces for the Deep Web based on attribute constraints

In order to query and retrieve the rich and useful information hidden in the Deep Web efficiently, extensive research on domain-specific Deep Web Data Integration Systems (DWDIS) has been carried out in recent years. In DWDIS, large-scale automatic integration of query interfaces of domain-specific Web Databases (WDBs) remains a serious challenge due to the scale of the problem and the great diversity of the WDBs' query interfaces. To address this challenge, in this paper, we first give a definition of the constraint matrix which can accurately describe three types of constraints (hierarchical constraints, group constraints and precedence constraints) and the strengths of attributes of a query interface, and then prove that the schema tree of the query interface corresponds to only one constraint matrix, and vice versa. Furthermore, we transform the problem of integrating domain-specific query interfaces into a problem of integrating the constraint matrices and set up a multi-objective optimization problem model. To effectively solve the optimization model, some strategies to extend and merge the constraint matrices are designed. A method for automatically detecting and filtering abnormal data (noises) in the query interfaces is also proposed. More importantly, a novel and efficient algorithm applicable to large-scale automatic integration of domain-specific query interfaces is developed. Finally, the proposed algorithm is evaluated by experiments on the real query interface data set. Our theoretical analysis and experimental results show that the proposed algorithm outperforms existing state-of-the-art integration algorithms of domain-specific query interfaces.

Efficient Approach for Knowledge Management Using Deep Web Information Retrieval System

Along with the development of network technology, web information are rapidly growing, and the way of information storage is gradually changed from the html to the database, thus web information can be divided into the surface web and deep web. Deep web is a concept corresponding to the surface web. It means from ordinary search engine that is difficult to discover the information content of a web page. The traditional crawler only crawl the content on the surface of a web, which makes the current traditional search engine, did not retrieve deep web data. Deep web compared with surface web has the advantage of large volume, high quality, theme single-minded, good structured. In view of several advantages, the establishment of deep web data integration system is becoming a research hotspot. The deep web query interface is the only entrance of the background database, so how to determine which web form is the query interface is important to the deep web information access. However, because the page proportion on the internet which contains querying interface is very small, using the traditional breadth-first strategy and keyword filtering method to crawl, it will download a lot of unrelated pages, spend a lot of resources, we need a way to efficiently find and collect the query interfaces through deep web crawling strategy. We proposed novel query planning approach, for executing different types of complex attribute through queries over multiple inter-dependent deep web data sources. increase accelerate query searching based on attribute selection, execution and propose optimization techniques, including query plan merging and grouping optimization. Keywords: Novel query planning approach, Deep web, Semantic Deep Web, Ontologies, attribute.

WETSUIT

We demonstrate a new powerful mashup tool called WETSUIT (Web EnTity Search and fUsIon Tool) to search and integrate web data from diverse sources and domain-specific entity search engines. WETSUIT supports adaptive search strategies to query sets of relevant entities with a minimum of communication overhead. Mashups can be composed using a set of high-level operators based on the Java-compatible language Scala. The operator implementation supports a high degree of parallel processing, in particular a streaming of entities between all data transformation operations facilitating a fast presentation of intermediate results. WETSUIT has already been applied to solve challenging integration tasks from different domains.

Gauging triple stores with actual biological data

BackgroundSemantic Web technologies have been developed to overcome the limitations of the current Web and conventional data integration solutions. The Semantic Web is expected to link all the data present on the Internet instead of linking just documents. One of the foundations of the Semantic Web technologies is the knowledge representation language Resource Description Framework (RDF). Knowledge expressed in RDF is typically stored in so-called triple stores (also known as RDF stores), from which it can be retrieved with SPARQL, a language designed for querying RDF-based models. The Semantic Web technologies should allow federated queries over multiple triple stores. In this paper we compare the efficiency of a set of biologically relevant queries as applied to a number of different triple store implementations.ResultsPreviously we developed a library of queries to guide the use of our knowledge base Cell Cycle Ontology implemented as a triple store. We have now compared the performance of these queries on five non-commercial triple stores: OpenLink Virtuoso (Open-Source Edition), Jena SDB, Jena TDB, SwiftOWLIM and 4Store. We examined three performance aspects: the data uploading time, the query execution time and the scalability. The queries we had chosen addressed diverse ontological or biological questions, and we found that individual store performance was quite query-specific. We identified three groups of queries displaying similar behaviour across the different stores: 1) relatively short response time queries, 2) moderate response time queries and 3) relatively long response time queries. SwiftOWLIM proved to be a winner in the first group, 4Store in the second one and Virtuoso in the third one.ConclusionsOur analysis showed that some queries behaved idiosyncratically, in a triple store specific manner, mainly with SwiftOWLIM and 4Store. Virtuoso, as expected, displayed a very balanced performance - its load time and its response time for all the tested queries were better than average among the selected stores; it showed a very good scalability and a reasonable run-to-run reproducibility. Jena SDB and Jena TDB were consistently slower than the other three implementations. Our analysis demonstrated that most queries developed for Virtuoso could be successfully used for other implementations.

A Personalization Recommendation Method Based on Deep Web Data Query

Deep Web is becoming a hot research topic in the area of database. Most of the existing researches mainly focus on Deep Web data integration technology. Deep Web data integration can partly satisfy people's needs of Deep Web information search, but it cannot learn users’ interest, and people search the same content online repeatedly would cause much unnecessary waste. According to this kind of demand, this paper introduced personalization recommendation to the Deep Web data query, proposed a user interest model based on fine-grained management of structured data and a similarity matching algorithm based on attribute eigenvector in allusion to personalization recommendation. Secondly, As for Deep Web information crawl, a crawl technology based on the tree structure is presented, with the traversal method of tree to solve the information crawl problems in the personalization service distributed in various web databases. Finally, developed a prototype recommendation system based on recruitment information, verified the efficiency and effectiveness of the personalization recommendation and the coverage and cost of Deep Web crawl through the experiment.

A Framework for Integrating, Exploring, and Searching Location-Based Web Data

This article presents the adaptation of a general search computing framework for exploratory search over Web data as suggested by the specificity of location-based data services. The result is a conceptual model of geographic entities, the spatial functions operating on them, and a special-purpose exploratory interface that lets users search combinations of georeferenced objects directly on a map. Such modifications help the general framework provide ranked extraction of relevant objects and their combinations, custom ranking functions, and cost-based access to location-based services.

Sample Web Database based on Probability & Statistics Model

In order to improve the efficiency of web data integration system, we need to collect more characteristics of web database, so that we can select some query-related web databases to join the query process and then decrease the number of web database and the number of query request. It is obvious that we can’t get the characteristics online because it is too inefficient. Sampling web database is a validate way to resolve the problem. This paper proposed web database sampling method based on probability & statistics model. First of all, we gave the expression method of web databases’ characteristics, and this was the foundation of our work. Secondly, we construct an initial query by the domain feature words. Thirdly, by analyzing the query result, we gave the method of constructing next query based on previous query result. At last, the method was stopped by judging if the similarity was stable or not. Experiment showed that this method had very good effect.

Linked Open Data Perspectives: Incorporating Linked Open Data into Information Extraction on the Web

Abstract Currently, the World Wide Web can be divided into two separate fields. The traditional Web of Documents consisting of hyperlinked web documents and the emerging Web of Data consisting of linked open data. We present ontology-based information extraction as core technology for bridging the gap between both fields. Based on this, we list three basic applications that integrate web data to web documents. Our SCOOBIE system can extract information of a linked open dataset mentioned as textual phrases in web documents. SCOOBIE returns machine interpretable metadata summarizing the content of a web document from the perspective of a linked open dataset. Based on SCOOBIE we present EPIPHANY, a system that returns extracted metadata back to the originating web document in form of semantic annotations. This allows users to request the Web of Data for more information about annotated subjects inside the web document. STERNTALER is a system that analyses extracted metadata from search results of a search engine. It generates semantic filters filled with facets of things that were extracted from web documents inside search results. This allows users filtering those web documents that contain information about specific subjects and facets.

A Novel Technique for Data Extraction from Hidden Web Databases

The large amount of information on web is stored in backend databases which are not indexed by traditional search engines. Such databases are referred to as Hidden web databases and extraction of this hidden web content is a potential research area as the pages are dynamically created through search query interfaces. However, direct query through this search interface is laborious way to search. Hence, there has been increased interest in retrieval and integration of hidden web data with a view to give high quality information to the web user. This paper proposes a novel approach that identifies Web page templates and the tag structures of a document in order to extract structured data from hidden web sources as the results returned in response to a user query are typically presented using template generated Web pages.

AN EFFICIENT WEB-BASED WRAPPER AND ANNOTATOR FOR TABULAR DATA

In the last few years, several works in the literature have addressed the problem of data extraction from web pages. The importance of this problem derives from the fact that, once extracted, data can be handled in a way similar to instances of a traditional database, which in turn can facilitate application of web data integration and various other domain specific problems. In this paper, we propose a novel table extraction technique that works on web pages generated dynamically from a back-end database. The proposed system can automatically discover table structure by relevant pattern mining from web pages in an efficient way, and can generate regular expression for the extraction process. Moreover, the proposed system can assign intuitive column names to the columns of the extracted table by leveraging Wikipedia knowledge base for the purpose of table annotation. To improve accuracy of the assignment, we exploit the structural homogeneity of the column values and their co-location information to weed out less likely candidates. This approach requires no human intervention and experimental results have shown its accuracy to be promising. Moreover, the wrapper generation algorithm works in linear time.

Utility Maximization Model for Deep Web Source Selection and Integration

The World Wide Web is witnessing an increase in the amount of structured content--vast collection of structured data are on the rise due to the deep web. Such Internet-scale deep web data integration tasks are becoming increasingly more common. In Internet-scale deep web data integration tasks, a primary challenge is to determine in which web database to be include d in the integration system. This paper presents a utility maximization model for resources selection of deep web data integration. This new model shows an efficient and effective way to estimate the approximate utility of the web database bringing to a given status of an integration system by integrating it. The utility of the web databases is synthesized by positive and negative utility. With the estimated utility information, web database selection can be made by explicitly optimizing the goal of high-utility(include as m uch and important data as possible in the selected databases, and the query cost of which as low as possible) in an iterative manner, where web databases are integrated incrementally. We experimentally demonstrate that our approach is efficient and finding high-utility data integration solutions.

A Novel Method for Extracting Information from Web Pages with Multiple Presentation Templates

Web information extraction is the key part of web data integration. With the need of e-commerce website and the development of web design, web pages with multiple presentation templates arise. The current web information extraction systems are usually based on single presentation template, so web pages with multiple presentation templates can’t be extracted efficiently. This paper focuses on the extraction problem about web pages with multiple presentation templates. Four different kinds of this problem have been considered, and a novel method based on path entropy, presentation regularity and ontology knowledge is presented. The experiment indicates that this method is very promising and it achieves excellent recall and precision.

Integrating Personal Web Data through Semantically Enhanced Web Portal

Currently, the World Wide Web is mostly composed of isolated and loosely connected islands. Connecting them together and retrieving only the information that is of interest to the user is the common Web usage process. Creating infrastructure that would support automation of that process by aggregating and integrating Web data in accordance to user's personal preferences would greatly improve today's Web usage. A significant part of Web data is available only through the login and password protected applications. As that data is very important for the usefulness of described process, proposed infrastructure needs to support authorized access to user's personal data. In this paper we propose a semantically enhanced Web portal that presents unique personalized user's entry to the domain-specific Web information. We also propose an identity management system that supports authorized access to the protected Web data. To verify the proposed solution, we have built Sweb - a semantically enhanced Web portal that uses proposed identity management system.

A Graph-Based Approach for Web Database Sampling

Web数据库中,海量的信息隐藏在具有特定查询能力的查询接口后面,使人无法了解一个Web数据库内容的特征,比如主题的分布、更新的频率等,这就为Deep Web数据集成带来了巨大的挑战.为了解决这个问题,提出了一种基于图模型的Web数据库采样方法,可以通过查询接口从Web数据库中以增量的方式获取近似随机的样本,即每次查询获取一定数量的样本记录,并且利用已经保存在本地的样本记录生成下一次的查询.该方法的一个重要特点是不受查询接口中属性表现形式的局限,因此是一种一般的Web数据库采样方法.在本地的模拟实验和真实Web数据库上的大量实验表明,该方法可以在较小代价下获得高质量的样本.;A flood of information is hidden behind the Web-based query interfaces with specific query capabilities, which makes it difficult to capture the characteristics of the Web database, such as the topic and the frequency of updates. This poses a great challenge for Deep Web data integration. To address this problem, a graph-based approach WDB-Sampler for Web database sampling is proposed in this paper, which can incrementally obtain sample records from a Web database through its query interface. That is, a number of samples are obtained for the current query, and one of them is transformed into the next query. The important characteristic of this approach is it can adapt to different kinds of attributes on the query interfaces. The extensive experiments on the local simulation Web databases and the real Web databases prove that the approach can achieve high-quality samples from a Web database at a lower cost.

Enabling global price comparison through semantic integration of web data

Sell and shop globally has been seen as a potential benefit of electronic business. A step toward realising this benefit is to know how things are selling worldwide. Global price comparison services would address this need. But there have not been many such services. We use a global price dispersion case study to illustrate the value of such services. Then we identify technology challenges, including semantic heterogeneity, in providing such services. We propose a mediation architecture to address the semantic heterogeneity problem, and demonstrate its feasibility with a prototype that enables global price comparison using web sources in several countries.

Enabling Global Price Comparison through Semantic Integration of Web Data

Sell Globally and Shop Globally have been seen as a potential benefit of web-enabled electronic business. One important step toward realizing this benefit is to know how things are selling in various parts of the world. A global price comparison service would address this need. But there have not been many such services. In this paper, we use a case study of global price dispersion to illustrate the need and the value of a global price comparison service. Then we identify and discuss several technology challenges including semantic heterogeneity, in providing a global price comparison service. We propose a mediation architecture to address the semantic heterogeneity problem, and demonstrate the feasibility of the proposed architecture by implementing a prototype that enables global price comparison using data from web sources in several countries.

A high performance integrated web data warehousing

Over the years, we have seen a significant number of integration techniques for data warehouses to support web integrated data. However, the existing works focus extensively on the design concept. ...

A high performance integrated web data warehousing

Over the years, we have seen a significant number of integration techniques for data warehouses to support web integrated data. However, the existing works focus extensively on the design concept. In this paper, we focus on the performance of a web database application such as an integrated web data warehousing using a well-defined and uniform structure to deal with web information sources including semi-structured data such as XML data, and documents such as HTML in a web data warehouse system. By using a case study, our implementation of the prototype is a web manipulation concept for both incoming sources and result outputs. Thus, the system not only can be operated through the web, it can also handle the integration of web data sources and structured data sources. Our main contribution is the performance evaluation of an integrated web data warehouse application which includes two tasks. Task one is to perform a verification of the correctness of integrated data based on the result set that is retrieved from the web integrated data warehouse system using complex and OLAP queries. The result set is checked against the result set that is retrieved from the existing independent data source systems. Task two is to measure the performance of OLAP or complex query by investigating source operation functions used by these queries to retrieve the data. The information of source operation functions used by each query is obtained using the TKPROF utility.