Structured Web Pages Research Articles

Web Page Ranking Based on Text Content and Link Information Using Data Mining Techniques

Thanks to the rapid expansion of the Internet, anyone can now access a vast array of information online. However, as the volume of web content continues to grow exponentially, search engines face challenges in delivering relevant results. Early search engines primarily relied on the words or phrases found within web pages to index and rank them. While this approach had its merits, it often resulted in irrelevant or inaccurate results. To address this issue, more advanced search engines began incorporating the hyperlink structures of web pages to help determine their relevance. While this method improved retrieval accuracy to some extent, it still had limitations, as it did not consider the actual content of web pages. The objective of the work is to enhance Web Information Retrieval methods by leveraging three key components: text content analysis, link analysis, and log file analysis. By integrating insights from these multiple data sources, the goal is to achieve a more accurate and effective ranking of relevant web pages in the retrieved document set, ultimately enhancing the user experience and delivering more precise search results the proposed system was tested with both multi-word and single-word queries, and the results were evaluated using metrics such as relative recall, precision, and F-measure. When compared to Google’s PageRank algorithm, the proposed system demonstrated superior performance, achieving an 81% mean average precision, 56% average relative recall, and a 66% F-measure.

A Framework for Automated Scraping of Structured Data Records From the Deep Web Using Semantic Labeling

The intent of this research is to come up with an automated web scraping system which is capable of extracting structured data records embedded in semi-structured web pages. Most of the automated extraction techniques in the literature captures repeated pattern among a set of similarly structured web pages, thereby deducing the template used for the generation of those web pages and then data records extraction is done. All of these techniques exploit computationally intensive operations such as string pattern matching or DOM tree matching and then perform manual labeling of extracted data records. The technique discussed in this paper departs from the state-of-the-art approaches by determining informative sections in the web page through repetition of informative content rather than syntactic structure. From the experiments, it is clear that the system has identified data rich region with 100% precision for web sites belonging to different domains. The experiments conducted on the real world web sites prove the effectiveness and versatility of the proposed approach.

Automated Discovery of Network Cameras in Heterogeneous Web Pages

Reduction in the cost of Network Cameras along with a rise in connectivity enables entities all around the world to deploy vast arrays of camera networks. Network cameras offer real-time visual data that can be used for studying traffic patterns, emergency response, security, and other applications. Although many sources of Network Camera data are available, collecting the data remains difficult due to variations in programming interface and website structures. Previous solutions rely on manually parsing the target website, taking many hours to complete. We create a general and automated solution for aggregating Network Camera data spread across thousands of uniquely structured web pages. We analyze heterogeneous web page structures and identify common characteristics among 73 sample Network Camera websites (each website has multiple web pages). These characteristics are then used to build an automated camera discovery module that crawls and aggregates Network Camera data. Our system successfully extracts 57,364 Network Cameras from 237,257 unique web pages.

Generating and selecting resilient and maintainable locators for Web automated testing

SummaryWeb user interface (UI) test automation strategies have been dominated by programmable and record–playback approaches. Of these, record–playback allows creating automation tests easily and reduces the cost of test generation. However, this approach increases the cost of test maintenance due to its unstable generated locators for identifying UI objects during playback. In this paper, we propose a new approach to generating and selecting resilient and maintainable locators. Our approach consists of two parts, a new XPath construction method and selecting the best XPath to locate the target element. Our XPath construction method relies on semantic structures of Web pages to locate the target element using its neighbors. We conducted an experiment on 15 popular websites. The results show that our approach outperforms the state‐of‐the‐practice/art Selenium IDE and Robula+ in locating target elements by effectively avoiding wrong locators. It also produces more readable XPaths (hence more maintainable tests) than do these approaches.

Layout Transposition for Non-Visual Navigation of Web Pages by Tactile Feedback on Mobile Devices.

In this paper, we present the results of an empirical study that aims to evaluate the performance of sighted and blind people to discriminate web page structures using vibrotactile feedback. The proposed visuo-tactile substitution system is based on a portable and economical solution that can be used in noisy and public environments. It converts the visual structures of web pages into tactile landscapes that can be explored on any mobile touchscreen device. The light contrasts overflown by the fingers are dynamically captured, sent to a micro-controller, translated into vibrating patterns that vary in intensity, frequency and temperature, and then reproduced by our actuators on the skin at the location defined by the user. The performance of the proposed system is measured in terms of perception of frequency and intensity thresholds and qualitative understanding of the shapes displayed.

A Method for Filtering Pages by Similarity Degree based on Dynamic Programming

To obtain the target webpages from many webpages, we proposed a Method for Filtering Pages by Similarity Degree based on Dynamic Programming (MFPSDDP). The method needs to use one of three same relationships proposed between two nodes, so we give the definition of the three same relationships. The biggest innovation of MFPSDDP is that it does not need to know the structures of webpages in advance. First, we address the design ideas with queue and double threads. Then, a dynamic programming algorithm for calculating the length of the longest common subsequence and a formula for calculating similarity are proposed. Further, for obtaining detailed information webpages from 200,000 webpages downloaded from the famous website “www.jd.com”, we choose the same relationship Completely Same Relationship (CSR) and set the similarity threshold to 0.2. The Recall Ratio (RR) of MFPSDDP is in the middle in the four filtering methods compared. When the number of webpages filtered is nearly 200,000, the PR of MFPSDDP is highest in the four filtering methods compared, which can reach 85.1%. The PR of MFPSDDP is 13.3 percentage points higher than the PR of a Method for Filtering Pages by Containing Strings (MFPCS).

Exploiting Multi-Category Characteristics and Unified Framework to Extract Web Content

Extracting web content is to obtain the required data embedded in web pages, usually including structured records, such as product information, and text content, such as news. Web pages use a large number of HTML tags to organize and to present various information. Both knowing little about the structures of web pages and mixing kinds of information in web pages are making the extraction process very challenging to guarantee extraction performance and extraction adaptability. This study proposes a unified web content extraction framework that can be applied in various web environments to extract both structured records and text content. First, we construct a characteristic container to hold kinds of characteristics related with extraction objectives, including visual text information, content semantics(instead of HTML tag semantics), web page structures, etc. Second, the above characteristics are integrated into an extraction framework for extraction decisions on different web sites. Especially, we put forward different strategies, path aggregation for extracting text content and HMM model for structured records, to locate the extraction area by exploiting both those extraction characteristics. Comparative experiments on multiple web sites with popular extraction methods, including CETR, CETD and CNBE, show that our proposed extraction method can provide better extraction precision and extraction adaptability.

Automated scraping of structured data records from health discussion forums using semantic analysis

Abstract The amount of information available in the Internet has an exponential growth and therefore, obtaining appropriate information from such a huge repository is an indispensable yet complicated task. As the structuring of web pages is diverse across websites, there is no “one size fits all” technique to perform web data extraction. It results in the need for devising a technique that is independent of structuring of web pages, which is addressed in this paper by identifying informative content through semantic analysis rather than syntactic structure. Social web forums contain web pages which are generated using server-side templates and the information present in such websites has wide variety of applications like opinion mining, sentiment analysis, topic modeling, trend analysis etc. Of the social media forums, health discussion forums play a crucial role and analyzing data extracted from such medical forums find its application in disease detection based on symptoms, determining adverse drug reactions, suggestion of clinical tests for diseases and so on. In this paper, a fully automated technique for extracting posts from various Medical Forum Websites has been devised and it performs well for differently structured web pages belonging to diverse forum websites. Since, the technique is based on semantic features, it can be applied to other social web forums as well.

Reordering Webpage Objects for Optimizing Quality-of-Experience

The quality of experience (QoE) perceived by users is a critical performance measure for Web browsing. “Above-The-Fold” (ATF) time has been recently recognized and widely used as a direct measure of user-end QoE by a number of studies. To reduce the ATF time, the existing works mainly focus on reducing the delay of networking. However, we observe that the webpage structures and content orders can also significantly affect theWeb QoE. In this paper, we propose a novel optimization framework that reorders the webpage objects to minimize the user-end ATF time. Our core idea is to first identify the webpage objects that consume the ATF time but have no impact on the page experience and then change the positions of these objects to achieve the minimum ATF time. We implement this framework and evaluate its performance with popular websites. The results show that the ATF time is greatly reduced compared with the existing works, especially for complex webpages.

Hviz: HTTP(S) traffic aggregation and visualization for network forensics

HTTP and HTTPS traffic recorded at the perimeter of an organization is an exhaustive data source for the forensic investigation of security incidents. However, due to the nested nature of today's Web page structures, it is a huge manual effort to tell apart benign traffic caused by regular user browsing from malicious traffic that relates to malware or insider threats. We present Hviz, an interactive visualization approach to represent the event timeline of HTTP and HTTPS activities of a workstation in a comprehensible manner. Hviz facilitates incident investigation by structuring, aggregating, and correlating HTTP events between workstations in order to reduce the number of events that are exposed to an investigator while preserving the big picture. We have implemented a prototype system and have used it to evaluate its utility using synthetic and real-world HTTP traces from a campus network. Our results show that Hviz is able to significantly reduce the number of user browsing events that need to be exposed to an investigator by distilling the structural properties of HTTP traffic, thus simplifying the examination of malicious activities that arise from malware traffic or insider threats.

Fast algorithms for finding a minimum repetition representation of strings and trees

A string with many repetitions can be represented compactly by replacing h-fold contiguous repetitions of a string r with (r)h. We present a compact representation, which we call a repetition representation (of a string) or RRS, by which a set of disjoint or nested tandem arrays can be compacted. In this paper, we study the problem of finding a minimum RRS or MRRS, where the size of an RRS is defined by the sum of the length of component letters and the description length of the component repetitions (⋅)h which is defined by wR(h) using a repetition weight function wR. We develop two dynamic programming-based algorithms to solve this problem: CMR, which works for any type of wR, and CMR-C, which is faster but can be applied to a constant wR only. CMR-C is an O(n2logn)-time O(nlogn)-space algorithm, which is more efficient in both time and space than CMR by a ((logn)/n)-factor, where n is the length of the given string. The problem of finding an MRRS for a string can be extended to that of finding a minimum repetition representation (of a tree) or MRRT for a given labeled ordered tree. For this problem, we present two algorithms, CMRT and CMRT-C, by using CMR and CMR-C, respectively, as a subroutine. As well as the theoretical analysis, we confirmed the efficiency of the proposed algorithms by experiments, which consist of the following three parts: First we demonstrated that CMR-C and CMRT-C are fast enough for large-scale data by using synthetic strings and trees, respectively. The size of an MRRS for a given string can be a measure of how compactly the string can be represented, meaning how well the string is structurally organized. This is also true of trees. To check such ability of MRRS-size, second we measured the size of an MRRS for chromosomes of nine different species. We found that all the chromosomes of the same species have a similar compression rate when realized by an MRRS. Run length encoding (RLE) was also shown to have species-specific compression rate, but species were separated more clearly by MRRS than by RLE. Third we examined the size of an MRRT for web pages of world-leading companies by using the tag trees, showing a consistency between the compression rate by an MRRT and visual web page structures.

Post-processing of Deep Web Information Extraction Based on Domain Ontology

Many methods are utilized to extract and process query results in deep Web, which rely on the different structures of Web pages and various designing modes of databases. However, some ...

HWPDE: Novel Approach for Data Extraction from Structured Web Pages

Diving into the World Wide Web for the purpose of fetching precious stones (relevant information) is a tedious task under the limitations of current diving equipments (Current Browsers). While a lot of work is being carried out to improve the quality of diving equipments, a related area of research is to devise a novel approach for mining. This paper describes a novel approach to extract the web data from the hidden websites so that it can be used as a free service to a user for a better and improved experience of searching relevant data. Through the proposed method, relevant data (Information) contained in the web pages of hidden websites is extracted by the crawler and stored in the local database so as to build a large repository of structured and indexed and ultimately relevant data. Such kind of extracted data has a potential to optimally satisfy the relevant Information starving end user.

Manufacturing Deep Web Service Management: Exploring Semantic Web Technologies

Since the wide adoption of database and Web service technologies in cross-enterprise manufacturing collaboration, leveraging deep Web contents through service discovery has become one of the most advanced trends in the manufacturing field. However, the effective service discovery and knowledge retrieval from manufacturing deep Web services is still a critical issue because of the underlying intricate structures of deep Web pages. This motivates us to seek a different way to manage manufacturing deep Web services by semantically annotating them to reflect their hidden, dynamic, and heterogeneous contents. In this article, a novel ontology-based approach that uses semantic Web paradigm is proposed, and its implementation in real applications is validated.

Performance Evaluation of Java Web Services: A Developer’s Perspective

As the population of the Internet grows rapidly the development of web technologies becomes extremely important. For the evolvement of Web 2.0, web services are essential. Web services are programs that allow computers of different platforms on the web to interactively communicate without the need of extra data for human reading interfaces and formats such as web page structures. Since web service is a future trend for the growth of internet, the tools that are used for development is also important. Although there are many choices of web service frameworks to choose from, developers should choose the framework that fits best to their application based on performance, time and effort for the framework. In this project, we chose four common frameworks to compare them in both qualitative and quantitative metrics. After running the tests, the results are statistically analyzed by SAS.

Closing the Loop in Webpage Understanding

The two most important tasks in information extraction from the Web are webpage structure understanding and natural language sentences processing. However, little work has been done toward an integrated statistical model for understanding webpage structures and processing natural language sentences within the HTML elements. Our recent work on webpage understanding introduces a joint model of Hierarchical Conditional Random Fields (HCRFs) and extended Semi-Markov Conditional Random Fields (Semi-CRFs) to leverage the page structure understanding results in free text segmentation and labeling. In this top-down integration model, the decision of the HCRF model could guide the decision making of the Semi-CRF model. However, the drawback of the top-down integration strategy is also apparent, i.e., the decision of the Semi-CRF model could not be used by the HCRF model to guide its decision making. This paper proposed a novel framework called WebNLP, which enables bidirectional integration of page structure understanding and text understanding in an iterative manner. We have applied the proposed framework to local business entity extraction and Chinese person and organization name extraction. Experiments show that the WebNLP framework achieved significantly better performance than existing methods.

Automatically Integrating Heterogeneous Ontologies from Structured Web Pages

This article presents an automated approach to integrate multiple analogous ontologies extracted from structured web pages into a common ontology. These ontologies from heterogeneous systems exhibit rich diversity in appearances, structures, terminologies and granularities. We design a unified similarity paradigm that can collect the implicit and explicit evidences that exhibit coherences among ontology and instance, semantic and structure, as well as linguistic and syntactic features. The similarity between ontology elements is derived from three aspects such as intension, extension and context, denoted by , where INT and EXT include corresponding weighted contents from their offspring, and CXT is relevant to evidences shown in their ancestors. The similarity in each aspect is calculated by means of their semantic overlapping and syntactic comparability. We develop a top-down matching algorithm based on matching space selection and similarity reuse; the algorithm facilitates less error-prone map-pings and lower computational cost.

Assessing metacognitive knowledge in web-based CALL: a neural network approach

The assessment of learners’ metacognitive knowledge level is crucial when developing computer-assisted language learning systems. Currently, many systems assess learners’ metacognitive knowledge level with pre-instructional questionnaires or metacognitive interviews. However, learners with limited language proficiency may be at a disadvantage in responding to verbal-report interview or questionnaire probes. The goal of this study is to present a neural network model that assesses automatically the learner’s metacognitive knowledge level by observing his/her online browsing behavior. The model is implemented through a multi-layer feed forward neural network. An experiment was conducted to examine the suitability of this model in different Web page structures. One hundred and forty-six college students were categorized into three groups according to three Web page structures: networked, hierarchical, and linear. The experiment results verified the suitability of the proposed model, and the MSEs of assessment of the three groups showed no significant differences with respect to the Web page structures.