Document Object Model Tree Research Articles

WIERT: Web Information Extraction via Render Tree

Web information extraction (WIE) is a fundamental problem in web document understanding, with a significant impact on various applications. Visual information plays a crucial role in WIE tasks as the nodes containing relevant information are often visually distinct, such as being in a larger font size or having a brighter color, from the other nodes. However, rendering visual information of a web page can be computationally expensive. Previous works have mainly focused on the Document Object Model (DOM) tree, which lacks visual information. To efficiently exploit visual information, we propose leveraging the render tree, which combines the DOM tree and Cascading Style Sheets Object Model (CSSOM) tree, and contains not only content and layout information but also rich visual information at a little additional acquisition cost compared to the DOM tree. In this paper, we present WIERT, a method that effectively utilizes the render tree of a web page based on a pretrained language model. We evaluate WIERT on the Klarna product page dataset, a manually labeled dataset of renderable e-commerce web pages, demonstrating its effectiveness and robustness.

Dom Tree as the base for webpage content extraction: Review

Because of the fast advancement of internet technology in the last twenty years, which leads to a huge number of web pages that contain a massive amount of information in every domain, the volume of available information has been steadily expanding every minute, so the analyzing and extracting information from web pages is becoming increasingly crucial, add to that information in webpages in an unstructured or semi-structured format need to transform in a structured format. 
 Since it is hard to collect the information manually, scientists have devised a variety of methods to help extract information from different domains in an automatic way. the main information in web pages is mixed with a significant amount of unrelated information (noise) like advertisements, boxes with links to relevant material, boxes with photos or other media, top and/or side navigation bars, animated commercials, etc., effect on the performance of information extraction and web content analysis technologies. to eliminate the noise by using the Document Object Model (DOM) that can easily reach every tag in the structure of the webpages to extract the information or delete the noise.
 This article explores in-depth DOM tree-based approaches, such as HTML tags and the DOM tree, by reviewing works from 2011 to 2021 and comparing numerous elements comprehensively, including classifier methods, contribution, limitation, and evaluation metrics.

Analysis of Enterprise Social Media Intelligence Acquisition Based on Data Crawler Technology

Abstract There are generally hundreds of millions of nodes in social media, and they are connected to a huge social network through attention and fan relationships. The news is spread through this huge social network. This paper studies the acquisition technology of social media topic data and enterprise data. The topic positioning technology based on Sina meta search and topic related keywords is introduced, and the crawling efficiency of topic crawlers is analyzed. Aiming at the factors of diverse and variable webpage structure on the Internet, this paper proposes a new Web information extraction algorithm by studying the general laws existing in the webpage structure, combining DOM (Document Object Model) tree and DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm. Several links in the algorithm are introduced in detail, including Web page processing, DOM tree construction, segmented text content acquisition, and web content extraction based on the DBSCAN algorithm. The simulation results show that the intelligence culture, intelligence system, technology platform and intelligence organization ecological collaboration strategy under the extraction of DOM tree and DBSCAN information can improve the level of intelligence participation of all employees. There is a significant positive correlation between the level of participation and the level of the intelligence environment of all employees. According to the research results, the DOM tree and DBSCAN information proposed in this paper can extract the enterprise’s employee intelligence and the effective implementation of relevant collaborative strategies, which can provide guidance for the effective implementation of the employee intelligence.

AdHealth: a feasibility study to measure digital food marketing to adolescents through Facebook.

To test the feasibility of a browser extension to estimate the exposure of adolescents to (un)healthy food and beverage advertisements on Facebook and the persuasive techniques used to market these foods and beverages. A Chrome browser extension (AdHealth) was developed to automatically collect advertisements seen by participants on their personal Facebook accounts. Information was extracted and sent to a web server by parsing the Document Object Model tree representation of Facebook web pages. Key information retrieved included the advertisement type seen and duration of each ad sighting. The WHO-Europe Nutrient Profile Model was used to classify the healthiness of products advertised as permitted (healthy) or not permitted (unhealthy) to be advertised to children. Auckland, New Zealand. Thirty-four Facebook users aged 16-18 years. The browser extension retrieved 4973 advertisements from thirty-four participants, of which 204 (4 %) were food-related, accounting for 1·1 % of the exposure duration. Of those food advertisements, 98 % were classified as not permitted, and 33·7 and 31·9 %, respectively, of those featured promotional characters or premium offers. The mean rate of exposure to not permitted food was 4·8 (sd = 2·5) advertisements per hour spent on Facebook. Using a Chrome extension to monitor exposure to unhealthy food and beverage advertisements showed that the vast majority of advertisements were for unhealthy products, despite numerous challenges to implementation. Further efforts are needed to develop tools for use across other social media platforms and mobile devices, and policies to protect young people from digital food advertising.

A Detection Method for Phishing Web Page Using DOM-Based Doc2Vec Model

Detecting phishing web pages is a challenging task. The existing detection method for phishing web page based on DOM (Document Object Model) is mainly aiming at obtaining structural characteristics but ignores the overall representation of web pages and the semantic information that HTML tags may have. This paper regards DOMs as a natural language with Doc2Vec model and learns the structural semantics automatically to detect phishing web pages. Firstly, the DOM structure of the obtained web page is parsed to construct the DOM tree, then the Doc2Vec model is used to vectorize the DOM tree, and to measure the semantic similarity in web pages by the distance between different DOM vectors. Finally, the hierarchical clustering method is used to implement clustering of web pages. Experiments show that the method proposed in the paper achieves higher recall and precision for phishing classification, compared to DOM-based structural clustering method and TF-IDF-based semantic clustering method. The result shows that using Paragraph Vector is effective on DOM in a linguistic approach.

A multi-layer approach to opinion polarity classification using augmented semantic tree kernels

ABSTRACTTremendous increase in user-generated content (UGC) published over the web in the form of natural language has posed a formidable challenge to automated information extraction (IE) and content analysis (CA). Techniques based on tree kernels (TK) have been successfully used for modelling semantic compositionality in many natural language processing (NLP) applications. Essentially, these techniques obtain the similarity of two production rules based on exact string comparison between the peer nodes. However, semantically identical tree fragments are forbidden even though they can contribute to the similarity of two trees. A mechanism needs to be addressed that accounts for the similarity of rules with varied syntax and vocabulary holding knowledge that are relatively analogous. In this paper, a hierarchical framework based on document object model (DOM) tree and linguistic kernels that jointly address subjectivity detection, opinion extraction and polarity classification is addressed. The model proceeds in three stages: during first stage, the contents of each DOM tree node is analysed to estimate the complexity of vocabulary and syntax using readability test. In second stage, the semantic tree kernels extended with word embeddings are used to classify nodes containing subjective and objective content. Finally, the content returned to be subjective is further examined for opinion polarity classification using fine-grained linguistic kernels. The efficiency of the proposed model is demonstrated through a series of experiments being conducted. The results reveal that the proposed polarity-enriched tree kernel (PETK) results in better prediction performance compared to the conventional tree kernels.

Automatic content extraction and time-aware topic clustering for large-scale social network on cloud platform

In recent years, with the increase in users in social network, the social network has had the feature of big data. The large-scale social network has become an indispensable part in people’s life. However, the traditional data mining technology cannot suit the large-scale social network. Thus, it is urgent to develop a more suitable mining technology for the large-scale social network. In this section, a crawler model based on semantic analysis and spatial clustering is proposed firstly. Then, the content extraction model based on document object model tree is built to extract the target text information from the links fetched by the proposed crawler model. The similarities between textual information in different regions are computed to choose the important information. Moreover, a two-stage topic clustering model based on time information is presented. The time information is introduced into the similarity computation between two posts or clusters. The single-pass algorithm is improved and applied in different clustering stage to improve the clustering accuracy. Finally, the proposed algorithms are evaluated on Hadoop platform. The Hadoop platform can effectively reduce the computing time and improve the server quality of users in large-scale social network. Meanwhile, the experiments demonstrate that the proposed algorithms are suitable for the data processing in large-scale social network.

Improving Performance of DOM in Semi-structured Data Extraction using WEIDJ Model

<p>Web data extraction is the process of extracting user required information from web page. The information consists of semi-structured data not in structured format. The extraction data involves the web documents in html format. Nowadays, most people uses web data extractors because the extraction involve large information which makes the process of manual information extraction takes time and complicated. We present in this paper WEIDJ approach to extract images from the web, whose goal is to harvest images as object from template-based html pages. The WEIDJ (Web Extraction Image using DOM (Document Object Model) and JSON (JavaScript Object Notation)) applies DOM theory in order to build the structure and JSON as environment of programming. The extraction process leverages both the input of web address and the structure of extraction. Then, WEIDJ splits DOM tree into small subtrees and applies searching algorithm by visual blocks for each web page to find images. Our approach focus on three level of extraction; single web page, multiple web page and the whole web page. Extensive experiments on several biodiversity web pages has been done to show the comparison time performance between image extraction using DOM, JSON and WEIDJ for single web page. The experimental results advocate via our model, WEIDJ image extraction can be done fast and effectively.</p>

Hunting for DOM-Based XSS vulnerabilities in mobile cloud-based online social network

This article presents a runtime Document Object Model (DOM) tree generator and nested context-aware sanitization based framework that alleviates the DOM-based XSS vulnerabilities from the mobile cloud-based OSN. The frameworks executes in dual mode: offline and online. The offline mode captures all the traces of modules of web applications and transformed such traces into static DOM tree for the extraction of benign script nodes. The legitimate script content embedded in such nodes will be marked in the whitelist of scripts. The online mode detects the injection of untrusted script content in the DOM tree generated at runtime. This was done by usually matching the script content embedded in this DOM tree with the whitelist of script code generated at offline mode. Any deviation observed in the script content will be marked as the injection of malicious script content in the dynamically generated DOM tree. This mode also identifies the different context of malicious variables embedded in such scripts and consequently executes the process of nested context-sensitive sanitization on them. The prototype of our mobile cloud-based framework was developed in Java and integrated the functionality of its components on iCanCloud simulator by creating different virtual machines with their proper link-to-link connectivity. The experimental testing and performance evaluation of our work was carried out on the open source OSN websites that are integrated in the virtual cloud servers. Evaluation results revealed that our framework is capable enough to detect the injection of untrusted/malicious script in the dynamically generated DOM tree with very low rate of false positives, false negatives and suffer from acceptable performance overhead.

Ekstraksi Data pada Tabel dari Halaman Web Menggunakan Pohon Document Object Model

Data on the web page can be available in various formats, such as table. With the growing of web pages, the need to extract data from tables is increasing. Results of the extraction can be used for integration with other web tables or stored in a database. This study discusses the extraction of data from a table on a web page using a Document Object Model (DOM) tree. The initial step of this extraction process is to transform the HTML document into a DOM tree. Then, by applying search methods Depth First Search (DFS), part of the data in the table is extracted and stored in a CSV file. An engine has been developed using Visual Basic. The results show that the engine can automatically extract data from the table that has the following characteristics: the number of rows and columns are not limited, able to handle all of the table orientation layout, and able to handle tables that are merged cells.

Efficient Identification of Structural Relationships for XML Queries using Secure Labeling Schemes

XML emerged as a de-facto standard for data representation and information exchange over the World Wide Web. By utilizing document object model (DOM), XML document can be viewed as XML DOM tree. Nodes of an XML tree are labeled to uniquely identify every node by following a labeling scheme. This paper proposes a method to efficiently identify the two structural relationships namely document order (DO) and sibling relationship that exist between the XML nodes using two secure labeling schemes specifically enhanced Dewey coding (EDC) and secure Dewey coding (SDC). These structural relationships influence the performance of XML queries so they need to be identified in efficient time. This paper implements the method to identify DO and sibling relationship using EDC and SDC labels for various real-time XML documents. Experiment results show the identification of DO and sibling relationship using SDC labels performs better than EDC labels for processing XML queries.

A novel algorithm for extracting the user reviews from web pages

Extracting the user reviews in websites such as forums, blogs, newspapers, commerce, trips, etc. is crucial for text processing applications (e.g. sentiment analysis, trend detection/monitoring and recommendation systems) which are needed to deal with structured data. Traditional algorithms have three processes consisting of Document Object Model (DOM) tree creation, extraction of features obtained from this tree and machine learning. However, these algorithms increase time complexity of extraction process. This study proposes a novel algorithm that involves two complementary stages. The first stage determines which HTML tags correspond to review layout for a web domain by using the DOM tree as well as its features and decision tree learning. The second stage extracts review layout for web pages in a web domain using the found tags obtained from the first stage. This stage is more time-efficient, being approximately 21 times faster compared to the first stage. Moreover, it achieves a relatively high accuracy of 96.67% in our experiments of review block extraction.

Automatic Testing of AJAX Applications through Dynamic Analysis of User Interface State Change

The growing popularity and importance of web applications have been increasing continuously in recent years. Use of JAVASCRIPT and dynamic DOM (Document Object Model) manipulation on the client side of web applications is becoming a widespread approach for achieving rich interactivity and responsiveness in modern web applications. AJAX (Asynchronous JAVASCRIPT and XML) based web applications rely on asynchronous client-server communication and client-side runtime manipulation of the DOM tree. This not only makes them fundamentally different from traditional web applications but also make them more error prone and harder to test. The proposed method for testing automatically AJAX application is based on a crawler to infer a state-flow graph for all client-side user interface states of an AJAX application. Focus is on obtaining a model by “crawling” an AJAX application, automatically clicking buttons and other user interface elements. In order to recognize failures in executions, use of invariants are proposed [1]. These invariants can be generic (e.g., after any client-side change the DOM should remain W3C-compliant valid HTML) or application-specific (e.g., the home-button in any state should lead back to the starting state). General Terms AJAX, JAVASCRIPT, XML, DOM

Web Content Extraction by Integrating Textual and Visual Importance of Web Pages

Web page has huge information and the information in the Web pages is useful in real world applications. The additional contents in the Web page like links, footers, headers and advertisements may cause the content extraction to be complicated. Irrelevant content in the Web page is treated as noisy content. A method is necessary to extract the informative content and discard the noisy content from Web pages. An integration of textual and visual importance is used to extract the informative content from Web pages. Initially a Web page is converted in to DOM (Document Object Model) tree. For each node in the DOM tree, textual and visual importance is calculated. Textual importance and visual importance is combined to form hybrid density. Density sum is calculated and used in content extraction algorithm to extract the informative content from Web pages. Performance of Web content extraction is obtained by calculating precision, recall, f-measure and accuracy. KeywordsContent Extraction, Web content Mining, DOM tree, Vision based Page Segmentation.

A tree-based learning approach for document structure analysis and its application to web search

AbstractIn this paper, we study the problem of structural analysis of Web documents aiming at extracting the sectional hierarchy of a document. In general, a document can be represented as a hierarchy of sections and subsections with corresponding headings and subheadings. We developed two machine learning models: heading extraction model and hierarchy extraction model. Heading extraction was formulated as a classification problem whereas a tree-based learning approach was employed in hierarchy extraction. For this purpose, we developed an incremental learning algorithm based on support vector machines and perceptrons. The models were evaluated in detail with respect to the performance of the heading and hierarchy extraction tasks. For comparison, a baseline rule-based approach was used that relies on heuristics and HTML document object model tree processing. The machine learning approach, which is a fully automatic approach, outperformed the rule-based approach. We also analyzed the effect of document structuring on automatic summarization in the context of Web search. The results of the task-based evaluation on TREC queries showed that structured summaries are superior to unstructured summaries both in terms of accuracy and user ratings, and enable the users to determine the relevancy of search results more accurately than search engine snippets.

A DOM-based Anchor-Hop-T Method for Web Application Information Extraction

In order to implement the information fusion of electronic products, the widely adopted approach is to extract information from HTML structure of business Website with deeply data processing. However, modeling Web application is hard to be solved that the data in HTML is semi-formal which displayed as DOM (Document Object Model) tree when using XML schema to data analysis. How to understand and extract information is first to be researched. The general model Anchor-Hop considering the text property and label property is simple to handle this problem. Therefore, it has low effectiveness. This model is sensitive to the data of HTML structure, that if the website structure is slightly changed the issue of extraction accuracy is encountered. As a result, the extraction rules should be redefined because of the changed structure. In order to improve extraction efficiency, this paper proposed a DOM-based dynamic model Anchor-Hop-T information extraction model. The HTML tags including table, ol and ul can be searched and processed using XPath so that it is convenience to extract corresponding Anchor data block. Furthermore, the location of Hop point is considered as invariant, by which our new model based on Anchor and Hop point introduces more concepts for extracting information, such as Anchor data block, Anchor locating library and AH relevance value. Finally, we try to give out an experiment to demonstrate the applicability of our approach.

Semantic Based Text Block Segmentation Using WordNet

Text block segmentation plays an important role in image search systems. Web pages are segmented into blocks for getting text around the image, which will later be used in retrieval process based on user query keywords. This paper presents semantic based text block segmentation for image retrieval system. Text block segmentation is performed for image indexing using semantic relevance between text blocks. Before segmentation process, web page is broken down in DOM (document object model) tree structure, where tags are used as nodes in the tree. Relatedness between text blocks are computed based on semantic relevance between terms. Semantic relevance in this paper is computed based on the hierarchical structure of the words and their hierarchical information is obtained from Wordnet dictionary. Image search by semantic relevance hence improves the accuracy of the search process

Auto-Testing AJAX-Based Web Applications

Asynchronous JAVASCRIPT and XML (AJAX)-based web applications (web apps) are different from traditional applications as they rely on stateful asynchronous communication and runtime manipulation of the DOM (document object model) tree. In this paper, we try to validate a method proposed by Mesbah et al. (2012) for testing AJAX applications automatically using two case studies. We try to identify AJAX-specific errors. Our goal is to replicate their experiment on two different web applications and comment on the performance of their approach. We use ‘Crawljax’ (designed by Mesbah et al.) to create a state-flow graph. First, we check the DOM validity for every state and, second, we check the navigation of the states. Finally, we present the results from our case studies.

A change detection system for unordered XML data using a relational model

The dramatic increase in the evolution of XML data available on the Internet requires a change detection system to keep track of important changes occurring during their life time. In this paper, we introduce a novel approach of detecting changes between two versions of unordered XML data stored in a traditional relational database using approaches like XRel. Most of the existing work in the area of XML change detection is mainly focused on detecting changes between two versions of XML data by constructing their Document Object Model (DOM) trees and then comparing these two tree structures based on Longest Common Sequence (LCS) using minimum edit distances. The basic tree comparison approach is not efficient in handling large XML files due to the fact that (1) an equivalent XML DOM tree will be twice as large as the original document and (2) the entire trees of both versions have to be memory resident during the comparison process. These two issues are constrained by the available main memory. In addition, existing approaches fail to detect changes among versions of XML data stored in relational databases as reverse mapping is not loss-less. We propose an efficient algorithm (XRel_Change_SQL) for detecting unordered changes between two XML data files stored in XRel as the underlying relational data model, using Structured Query Language (SQL). We compare the efficiency and quality of our change detection algorithm with existing XML change detection tools like X-Diff, DeltaXML and XANDY. We provide an experimental evaluation of the results obtained from the benchmark datasets as well as some synthetic datasets to show that our approach is highly scalable, and results in a much better efficiency and delta quality than the aforementioned approaches and tools.

Automatically extracting user reviews from forum sites

User reviews in forum sites are the important information source for many popular applications (e.g., monitoring and analysis of public opinion), which are usually represented in form of structured records. To the best of our knowledge, little existing work reported in the literature has systemically investigated the problem of extracting user reviews from forum sites. Besides the variety of web page templates, user-generated reviews raise two new challenges. First, the inconsistency of review contents in terms of both the document object model (DOM) tree and visual appearance impair the similarity between review records; second, the review content in a review record corresponds to complicated subtrees rather than single nodes in the DOM tree. To tackle these challenges, we present WeRE — a system that performs automatic user review extraction by employing sophisticated techniques. The review records are extracted from web pages based on the proposed level-weighted tree similarity algorithm first, and then the review contents in records are extracted exactly by measuring the node consistency. Our experimental results based on 20 forum sites indicate that WeRE can achieve high extraction accuracy.