Wrapper Generation Research Articles

Automatic animal monitoring systems based on AutoML technologies

In contemporary agricultural contexts, the sector is experiencing active processes of automation, underscoring the need for effective tools to develop such automated systems. The utilization of automation tools enhances the efficiency of numerous processes, including those in the domain of animal monitoring. This article examines the application of the AutoML approach as a means for automating the process of generating deep learning models employed in automatic monitoring systems. The VGG19 architecture has been chosen as a testbed for demonstrating the capabilities of the developed technologies. This well-established architecture for deep learning models is designed for object recognition in images.The present study implements a technology for automated structural-parametric synthesis of VGG19 models and the optimization of their hyperparameters. Such an approach allows for the automated creation of models tailored to specific applied problems, even for users lacking specialized knowledge in deep learning.The system delineated in this work is developed on the AutoGenNet software platform, which embodies the No-Code development concept. This concept conceals complex aspects of model creation and training processes from users, significantly lowering the entry barrier for newcomers. Additionally, the AutoGenNet platform incorporates a mechanism for the automatic generation of software wrappers, facilitating efficient interaction with trained models.All aforementioned aspects have contributed to the effective implementation of the AutoML approach for automating the generation and training processes of the VGG19 model. Consequently, the processes associated with solving automatic monitoring tasks reliant on deep learning models have been significantly simplified and expedited.The developed system has been tested on the task of recognizing individual cows. Test results indicated that the system possesses a high degree of scalability and can be adapted for the automated generation of other object recognition models, thereby opening avenues for addressing a diverse array of applied challenges related to the monitoring of various animal species.

A New PyROOT: Modern, Interoperable and More Pythonic

Python is nowadays one of the most widely-used languages for data science. Its rich ecosystem of libraries together with its simplicity and readability are behind its popularity. HEP is also embracing that trend, often using Python as an interface language to access C++ libraries for the sake of performance. PyROOT, the Python bindings of the ROOT software toolkit, plays a key role here, since it allows to automatically and dynamically invoke C++ code from Python without the generation of any static wrappers beforehand. In that sense, this paper presents the efforts to create a new PyROOT with three main qualities: modern, able to exploit the latest C++ features from Python; pythonic, providing Python syntax to use C++ classes; interoperable, able to interact with the most important libraries of the Python data science toolset.

An Improved Wrapper Generation Using Self Organizing Maps and Meta Heuristic Technique for Web Based Biomedical Data Mining

An Improved Wrapper Generation Using Self Organizing Maps and Meta Heuristic Technique for Web Based Biomedical Data Mining

Introduction to the special issue of the World Wide Web journal on “Social Media Preservation and Applications”

Social media have become firmly established as a new publishing and communication tool with significant value for all the aspects of contemporary society, culture and science. Social media feature a dynamic and continuously evolving nature, fostering the creation and exchange of concepts and ideas in an unprecedented scale. The leading role of social media coupled with the inherent volatility and the ephemeral nature of this medium, compared with other forms of electronic communication, has rendered social media preservation as necessary. It is critical to preserve social media as they are an essential part of our heritage and they can prove valuable for current and future generations. Social media preservation is an advanced topic of digital preservation, encompassing the capture, management, preservation and availability of social media for future use and research. To achieve this goal, we need to advance the state of the art in a wide range of diverse topics such as social media modeling and analysis, classification, information retrieval, content migration, replication, emulation and web preservation strategies. Furthermore, we need to focus not only on theoretical advances but also on innovative applications and systems. In this special issue, we are aiming to exchange the latest fundamental advances in the state of the art and practice of social media preservation and important related areas. In this issue we have included 7 papers, which have been selected out of 17 papers that were submitted in response to our call. Some information about these papers follows. The paper by L. Akritidis and P. Bozanis entitled “Improving Opinionated Blog Retrieval Effectiveness with Quality Measures and Temporal Features” deals with the problem of retrieval of blog entries, which contain opinions about entries by paying special attention to the importance of such entries. The proposed ranking method takes into account the influence of the blogger who authored an opinion, the reputation of the specific blog site and the impact of the blog post itself. Experiments with TREC Blogs08 dataset shows enhancement of retrieval precision. The paper by G. Gkotsis, K. Stepanyan, A. Cristea and M. Joy entitled “Entropy-based Automated Wrapper Generation for Web Data Extraction” focus on the process of weblog data extraction. The approach includes a model for generating a wrapper that exploits web feeds for deriving probabilistically (based on entropy) a set of extraction rules in an automatic way. The World Wide Web (2014) 17:691–693 DOI 10.1007/s11280-014-0282-4

DWDE-IR: An Efficient Deep Web Data Extraction for Information Retrieval on Web Mining

Deep Web is a widely unexplored data source, is becoming an important research topic. Retrieving structured data from deep web pages is the challenging problem due to their complex structure. In this paper, Information extracts on the Deep Web pages based on the Deep Web Data Extraction technique (DWDR-IR). Search engines usually return a large number of pages in response to the user queries. To help the users to navigate in the result list, ranking methods are activated on the search results. In this paper, a page ranking mechanism called Coherence Ratio based Page (CRP) ranking algorithm is used. To retrieve the information accurately, an approach called WordNet is used. WordNet checks the similarity of data records and find the correct data region with higher precision using the semantic properties of data records. This concept is very important to display the valuable results occur on the top of the result list on the basis of browsing behavior of the user, it reduces the search space and provides high accuracy. This approach handles the visual features on the deep web data extraction, including data item extraction, data record extraction and visual wrapper generation. The proposed work removes all noise such as header, footer, irrelevant advertisement and irrelevant content using NoiSe Filter (NSFilter) algorithm. The proposed method retrieves perfect extraction of relevant results from the deep web pages. DWDE-IR results higher precision, recall and filter accuracy than the existing method ViDE.

Semisupervised Wrapper Choice and Generation for Print-Oriented Documents

Information extraction from printed documents is still a crucial problem in many interorganizational workflows. Solutions for other application domains, for example, the web, do not fit this peculiar scenario well, as printed documents do not carry any explicit structural or syntactical description. Moreover, printed documents usually lack any explicit indication about their source. We present a system, which we call PATO, for extracting predefined items from printed documents in a dynamic multisource scenario. PATO selects the source-specific wrapper required by each document, determines whether no suitable wrapper exists, and generates one when necessary. PATO assumes that the need for new source-specific wrappers is a part of normal system operation: new wrappers are generated online based on a few point-and-click operations performed by a human operator on a GUI. The role of operators is an integral part of the design and PATO may be configured to accommodate a broad range of automation levels. We show that PATO exhibits very good performance on a challenging data set composed of more than 600 printed documents drawn from three different application domains: invoices, datasheets of electronic components, and patents. We also perform an extensive analysis of the crucial tradeoff between accuracy and automation level.

Entropy-based automated wrapper generation for weblog data extraction

This paper proposes a fully automated information extraction methodology for weblogs. The methodology integrates a set of relevant approaches based on the use of web feeds and processing of HTML for the extraction of weblog properties. The approach includes a model for generating a wrapper that exploits web feeds for deriving a set of extraction rules automatically. Instead of performing a pairwise comparison between posts, the model matches the values of the web feeds against their corresponding HTML elements retrieved from multiple weblog posts. It adopts a probabilistic approach for deriving a set of rules and automating the process of wrapper generation. An evaluation of the model is conducted on a collection of weblogs reporting a prediction accuracy of 89 %. The results of this evaluation show that the proposed technique enables robust extraction of weblog properties and can be applied across the blogosphere.

Lexical Semantic based Bayesian Model for Adaptive Wrapper Generation

This paper focuses on an unsupervised information extraction system. Two kinds of features related to the text fragments from the Web documents are investigated. The first type of feature is called, a site-invariant feature. The second type of feature is called a site-dependent feature. Feature selection algorithm is used for wrapper generation from the site invariant and site-dependent information. The wrapper is generated and the new attribute is also discovered and adapted with wrapper by using the Bayesian learning method and E-M algorithm along with the lexical semantics search method. Our wrapper can be able to adapt with the new unseen sites. Our system efficiency is evaluated based on some performance measures and the effectiveness of the system is evaluated by using the performance metrics, precision, recall, f-measure, true positive and false positive in the real time web sites.

A Fuzzy Logic Approach to Wrapping PDF Documents

The PDF format represents the de facto standard for print-oriented documents. In this paper, we address the problem of wrapping PDF documents, which raises new challenges in several contexts of text data management. Our proposal is based on a novel bottom-up hierarchical wrapping approach that exploits fuzzy logic to handle the “uncertainty” which is intrinsic to the structure and presentation of PDF documents. A PDF wrapper is defined by specifying a set of group type definitions that impose a target structure to groups of tokens containing the required information. Constraints on token groupings are formulated as fuzzy conditions, which are defined on spatial and content predicates of tokens. We define a formal semantics for PDF wrappers and propose an algorithm for wrapper evaluation working in polynomial time with respect to the size of a PDF document. The proposed approach has been implemented in a wrapper generation system that offers visual capabilities to assist the designer in specifying and evaluating a PDF wrapper. Experimental results have shown good accuracy and applicability of our system to PDF documents of various domains.

AN EFFICIENT WEB-BASED WRAPPER AND ANNOTATOR FOR TABULAR DATA

In the last few years, several works in the literature have addressed the problem of data extraction from web pages. The importance of this problem derives from the fact that, once extracted, data can be handled in a way similar to instances of a traditional database, which in turn can facilitate application of web data integration and various other domain specific problems. In this paper, we propose a novel table extraction technique that works on web pages generated dynamically from a back-end database. The proposed system can automatically discover table structure by relevant pattern mining from web pages in an efficient way, and can generate regular expression for the extraction process. Moreover, the proposed system can assign intuitive column names to the columns of the extracted table by leveraging Wikipedia knowledge base for the purpose of table annotation. To improve accuracy of the assignment, we exploit the structural homogeneity of the column values and their co-location information to weed out less likely candidates. This approach requires no human intervention and experimental results have shown its accuracy to be promising. Moreover, the wrapper generation algorithm works in linear time.

A framework for collaborative working environments

To utilize the advantages of existing and emerging Internet techniques and to meet the demands for a new generation of collaborative working environments, a framework with an upperware–middleware architecture is proposed, which consists of four layers: resource layer, middleware layer, upperware layer and application layer. The upperware contains intelligent agents and plug/play facilities; the former coordinates and controls multiple middleware techniques such as Grid computing, Web-services and mobile agents, while the latter are used for the applications, such as semantic CAD, to plug and loose couple into the system. The method of migrating legacy software using automatic wrapper generation technique is also presented. A prototype mobile environment for collaborative product design is presented to illustrate the utilization of the CWE framework in collaborative design and manufacture.

WRAPPER INFERENCE FOR AMBIGUOUS WEB PAGES

Several studies have concentrated on the generation of wrappers for web data sources. As wrappers can be easily described as grammars, the grammatical inference heritage could play a significant role in this research field. Recent results have identified a new subclass of regular languages, called prefix mark-up languages, that nicely abstract the structures usually found in HTML pages of large web sites. This class has been proven to be identifiable in the limit, and a PTIME unsupervised learning algorithm has been previously developed. Unfortunately, many real-life web pages do not fall in this class of languages. In this article we analyze the roots of the problem and we propose a technique to transform pages in order to bring them into the class of prefix mark-up languages. In this way, we have a practical solution without renouncing to the formal background defined within the grammatical inference framework. We report on some experiments that we have conducted on real-life web pages to evaluate the approach; the results of this activity demonstrate the effectiveness of the presented techniques.

XWRAPComposer

We present a service-oriented architecture and a set of techniques for developing wrapper code generators, including the methodology of designing an effective wrapper program construction facility and a concrete implementation, called XWRAPComposer. Our wrapper generation framework has two unique design goals. First, we explicitly separate tasks of building wrappers that are specific to a Web service from the tasks that are repetitive for any service, thus the code can be generated as a wrapper library component and reused automatically by the wrapper generator system. Second, we use inductive learning algorithms that derive information flow and data extraction patterns by reasoning about sample pages or sample specifications. More importantly, we design a declarative rule-based script language for multi-page information extraction, encouraging a clean separation of the information extraction semantics from the information flow control and execution logic of wrapper programs. We implement these design principles with the development of the XWRAPComposer toolkit, which can semi-automatically generate WSDL-enabled wrapper programs. We illustrate the problems and challenges of multi-page data extraction in the context of bioinformatics applications and evaluate the design and development of XWRAPComposer through our experiences of integrating various BLAST services.

STAVIES: a system for information extraction from unknown Web data sources through automatic Web wrapper generation using clustering techniques

A fully automated wrapper for information extraction from Web pages is presented. The motivation behind such systems lies in the emerging need for going beyond the concept of "human browsing". The World Wide Web is today the main "all kind of information" repository and has been so far very successful in disseminating information to humans. By automating the process of information retrieval, further utilization by targeted applications is enabled. The key idea in our novel system is to exploit the format of the Web pages to discover the underlying structure in order to finally infer and extract pieces of information from the Web page. Our system first identifies the section of the Web page that contains the information to be extracted and then extracts it by using clustering techniques and other tools of statistical origin. STAVIES can operate without human intervention and does not require any training. The main innovation and contribution of the proposed system consists of introducing a signal-wise treatment of the tag structural hierarchy and using hierarchical clustering techniques to segment the Web pages. The importance of such a treatment is significant since it permits abstracting away from the raw tag-manipulating approach. Experimental results and comparisons with other state of the art systems are presented and discussed in the paper, indicating the high performance of the proposed algorithm.

Automatic information extraction from large websites

Information extraction from websites is nowadays a relevant problem, usually performed by software modules called wrappers. A key requirement is that the wrapper generation process should be automated to the largest extent, in order to allow for large-scale extraction tasks even in presence of changes in the underlying sites. So far, however, only semi-automatic proposals have appeared in the literature.We present a novel approach to information extraction from websites, which reconciles recent proposals for supervised wrapper induction with the more traditional field of grammar inference. Grammar inference provides a promising theoretical framework for the study of unsupervised---that is, fully automatic---wrapper generation algorithms. However, due to some unrealistic assumptions on the input, these algorithms are not practically applicable to Web information extraction tasks.The main contributions of the article stand in the definition of a class of regular languages, called the prefix mark-up languages, that abstract the structures usually found in HTML pages, and in the definition of a polynomial-time unsupervised learning algorithm for this class. The article shows that, differently from other known classes, prefix mark-up languages and the associated algorithm can be practically used for information extraction purposes.A system based on the techniques described in the article has been implemented in a working prototype. We present some experimental results on known Websites, and discuss opportunities and limitations of the proposed approach.

Automatic composite wrapper generation for semi-structured biological data based on table structure identification

Biological data analyses usually require complex manipulations involving tool applications, multiple web site navigation, result selection and filtering, and iteration over the internet. Most biological data are generated from structured databases and by applications and presented to the users embedded within repeated structures, or tables, in HTML documents. In this paper we outline a novel technique for the identification of table structures in HTML documents. This identification technique is then used to automatically generate composite wrappers for applications requiring distributed resources. We demonstrate that our method is robust enough to discover standard as well as non-standard table structures in HTML documents. Thus our technique outperforms contemporary techniques used in systems such as XWrap and AutoWrapper. We discuss our technique in the context of our PickUp system that exploits the theoretical developments presented in this paper and emerges as an elegant automatic wrapper generation system.

Wrapper Generation for Collecting Comparative Shopping Information

This paper proposes a wrapper generation method for collecting comparative shopping information from various Internet shopping malls. The proposed method is a kind of supervised learning method to learn wrappers from sample web pages along with information locations designated by the administrators. It generates wrappers expressed in the form of generalized tags sequences and frame filling procedures for semi-structured web pages. The paper also presents how to use the learned wrappers and describes a prototype system which implemented the proposed ideas and methods.

Querying websites using compact skeletons

Several commercial applications, such as online comparison shopping and process automation, require integrating information that is scattered across multiple websites or XML documents. Much research has been devoted to this problem, resulting in several research prototypes and commercial implementations. Such systems rely on wrappers that provide relational or other structured interfaces to websites. Traditionally, wrappers have been constructed by hand on a per-website basis, constraining the scalability of the system. We introduce a website structure inference mechanism called compact skeletons that is a step in the direction of automated wrapper generation. Compact skeletons provide a transformation from websites or other hierarchical data, such as XML documents, to relational tables. We study several classes of compact skeletons and provide polynomial-time algorithms and heuristics for automated construction of compact skeletons from websites. Experimental results show that our heuristics work well in practice. We also argue that compact skeletons are a natural extension of commercially deployed techniques for wrapper construction.

Wrapper Maintenance: A Machine Learning Approach

The proliferation of online information sources has led to an increased use of wrappers for extracting data from Web sources. While most of the previous research has focused on quick and efficient generation of wrappers, the development of tools for wrapper maintenance has received less attention. This is an important research problem because Web sources often change in ways that prevent the wrappers from extracting data correctly. We present an efficient algorithm that learns structural information about data from positive examples alone. We describe how this information can be used for two wrapper maintenance applications: wrapper verification and reinduction. The wrapper verification system detects when a wrapper is not extracting correct data, usually because the Web source has changed its format. The reinduction algorithm automatically recovers from changes in the Web source by identifying data on Web pages so that a new wrapper may be generated for this source. To validate our approach, we monitored 27 wrappers over a period of a year. The verification algorithm correctly discovered 35 of the 37 wrapper changes, and made 16 mistakes, resulting in precision of 0.73 and recall of 0.95. We validated the reinduction algorithm on ten Web sources. We were able to successfully reinduce the wrappers, obtaining precision and recall values of 0.90 and 0.80 on the data extraction task.

Wrapping of Soft IPs for Interface‐based Design Using Heterogeneous Metaprogramming

We discuss the application of heterogeneous metaprogramming (MPG) for the interface-based design, which deals with the synthesis of the communication interfaces between Intellectual Property components (IPs). Heterogeneous MPG is based on the simultaneous usage of a domain language for describing domain functionality and a metalanguage for developing generic components and program generators. We present a design framework based on the MPG techniques. The novelty of our approach is that we apply the concept of heterogeneous MPG for the automatic generation of IP wrappers for communication between the third-party soft IPs systematically. We present a case study for the synthesis of the communication interfaces using a FIFO protocol.