Structure Of Web Pages Research Articles

SIMHAR - Smart Distributed Web Crawler for the Hidden Web Using SIM+Hash and Redis Server

Developing a distributed web crawler obliges major engineering challenges, all of which are eventually associated to scale. To retain corpus of search engine and a reasonable state of freshness, the crawler must be distributed over multiple computers. In distributed crawling, crawling agents are given a task to fetch and download web pages. The number and heterogeneous structure of web pages are increasing rapidly. This made the performance a serious challenge to web crawler systems. In this paper, a distributed web crawler for the hidden web is proposed and implemented. It combines and integrates, scrapy framework and Redis server. Crawling is split into three stages-adaption, relevant source selection and underlying content extraction. The crawler accurately detects and submit the searchable forms. Duplication detection is based on hybrid technology using hash-maps of Redis and Sim+Hash. Redis server is also acting as a data store for a massive amount of web data so that the growth of hidden web databases is handled ensuring scalability.

Translation of news reports related to COVID-19 of Japanese Linguistics based on page link mining

In the face of the current epidemic situation, news reports are facing the problem of higher accuracy. The speed and accuracy of public emergency news depends on the accuracy of web page links and tags clustering. An improved web page clustering method based on the combination of topic clustering and structure clustering is proposed in this paper. The algorithm takes the result of web page structure clustering as the weight factor. Combined with the web content clustering by K-means algorithm, the basic content that meets the conditions is selected. Through the improved translator of clustering algorithm, it is translated into Chinese and compared with the target content to analyze the similarity. It realized the translation aim of new crown virus epidemic related news report of Japanese Linguistics based on page link mining.

VB-PTC: Visual Block Multi-Record Text Extraction Based on Sensor Network Page Type Conversion

Usually, in addition to the main content, web pages contain additional information in the form of noise, such as navigation elements, sidebars and advertisements. This kind of noise has nothing to do with the main content, it will affect the tasks of data mining and information retrieval so that the sensor will be damaged by the wrong data and interference noise. Because of the diversity of web page structure, it is a challenge to detect relevant information and noise in order to improve the true reliability of sensor networks. In this paper, we propose a visual block construction method based on page type conversion (VB-PTC). This method uses a combination of site-level noise reduction based on hashtree and page-level noise reduction based on linked clusters to eliminate noise in web articles, and it successfully converts multi-record complex pages to multi-record simple pages, effectively simplifying the rules of visual block construction. In the aspect of multi-record content extraction, according to the characteristics of different fields, we use different extraction methods, combined with regular expression, natural language processing and symbol density detection methods which greatly improves the accuracy of multi-record content extraction. VB-PTC can be effectively used for information retrieval, content extraction and page rendering tasks.

A Pure Visual Approach for Automatically Extracting and Aligning Structured Web Data

Database-driven websites and the amount of data stored in their databases are growing enormously. Web databases retrieve relevant information in response to users’ queries; the retrieved information is encoded in dynamically generated web pages as structured data records. Identifying and extracting retrieved data records is a fundamental task for many applications, such as competitive intelligence and comparison shopping. This task is challenging due to the complex underlying structure of such web pages and the existence of irrelevant information. Numerous approaches have been introduced to address this problem, but most of them are HTML-dependent solutions that may no longer be functional with the continuous development of HTML. Although a few vision-based techniques have been introduced, various issues exist that inhibit their performance. To overcome this, we propose a novel visual approach, i.e., programming-language-independent, for automatically extracting structured web data. The proposed approach makes full use of the natural human tendency of visual object perception and the Gestalt laws of grouping. The extraction system consists of two tasks: (1) data record extraction, where we apply three of the Gestalt laws (i.e., laws of continuity, proximity, and similarity), which are used to group the adjacently aligned visually similar data records on a web page; and (2) data item extraction and alignment, where we employ the Gestalt law of similarity, which is utilized to group the visually identical data items. Our experiments upon large-scale test sets show that the proposed system is highly effective and outperforms the two state-of-art vision-based approaches, ViDE and rExtractor. The experiments produce an average F1 score of 86.02%, which is approximately 55% and 36% better than that of ViDE and rExtractor for data record extraction, respectively; and an average F1 score of 86.19%, which is approximately 39% better than that of ViDE for data item extraction.

Design of Beidou Satellite System in Ocean Logistics Real-Time Tracking System

Hao, Q.; Wang, Z.-F. and Qin, L.-L., 2019. Design of Beidou satellite system in ocean logistics real-time tracking system. In: Gong, D.; Zhu, H., and Liu, R. (eds.), Selected Topics in Coastal Research: Engineering, Industry, Economy, and Sustainable Development. Journal of Coastal Research, Special Issue No. 94, pp. 204–207. Coconut Creek (Florida), ISSN 0749-0208.In order to ensure the quality and safety of pelagic fish catches in the process of transportation, administrator needs to carry out real-time monitoring and management to the transport and quality status during this period and establish a complete management system. This paper puts forward that the real-time online monitoring of transport location, catches temperature, humidity, PH value and other physical property data can be used during the application process of Beidou System and RFID temperature tags in the cold chain logistics of ocean catches. The system consists of 3 parts, i.e. vehicle or vessel tracking, parameter monitoring and data management and analysis. The system applies the technologies such as Beidou, ZigBee and Internet as well as the Java-programming client web page structure and the traditional SSH2 as its code framework. Hence, it realizes the real-time monitoring of whole catches transportation logistics process and makes administrator timely know about the status of catches so as to ensure the product quality. This system can guarantee the quality, safety and transport efficiency of pelagic fish products and provides a support for the building of cold chain logistics system.

Expert Information Automatic Extraction for IOT Knowledge Base

With the rapid development of IOT technology, the requirement of effective and accurate retrieval of domain knowledge is growing. Automatically extract various information of expert from the massive web pages and generate a dynamic and wholeness profile model are important for knowledge base. However, the obvious differences in structure and content semantics of web pages between any two websites shows traditional web crawler are hard to understand the semantic of the web page and extract the critical information of expert. Therefore, a six-dimension expert profile model was introduced and then a sequence tagging method with LSTM-CRF model was proposed to automatically extract rich semantic information basing on organization structure, meaning of words and attributes of experts. The results of the experiment on test data sets illustrated that the precision rate and recall rate about the job experience and research field of experts are 67.8%, 66.6% and 82.4%, 79.6%, respectively. In addition, the overall average F value about some obvious features of expert, such as name, title, email, achievement, etc., reaches 82.5%, which is better than the results by MEMM and LSTM algorithms.

Research on link blocks recognition of web pages

The link block is a typical type of block structure of web pages; it is an important research object in the fields of web data mining. Firstly, block and block tree are proposed as the basic concepts of subsequent explorations, and then an approach of building block trees is put forward. Secondly, four rules for link block discrimination and two indicators for recognition results evaluation are put forward based on the concept of block. Finally, a strategy named forward algorithm for discovery of link block (FAD) is proposed and a corresponding experiment with different parameters is performed to verify the strategy. The results show that the FAD can be flexible to achieve recognition of link blocks under different granularity conditions. Concepts and approaches presented in this paper have a good prospect in the fields of web data processing such as advertising block recognition and web content extraction.

Research on link blocks recognition of web pages

The link block is a typical type of block structure of web pages; it is an important research object in the fields of web data mining. Firstly, block and block tree are proposed as the basic concepts of subsequent explorations, and then an approach of building block trees is put forward. Secondly, four rules for link block discrimination and two indicators for recognition results evaluation are put forward based on the concept of block. Finally, a strategy named forward algorithm for discovery of link block (FAD) is proposed and a corresponding experiment with different parameters is performed to verify the strategy. The results show that the FAD can be flexible to achieve recognition of link blocks under different granularity conditions. Concepts and approaches presented in this paper have a good prospect in the fields of web data processing such as advertising block recognition and web content extraction.

Web Data Extraction Approach for Deep Web using WEIDJ

Data extraction is one of the most prominent areas in data mining analysis that is been extensively studied especially in the field of data requirements and reservoir. The main aim of data extraction with regards to semi-structured data is to retrieve beneficial information from the World Wide Web. The data from large web data also known as deep web is retrievable but it requires request through form submission because it cannot be performed by any search engines. Data mining applications and automatic data extraction are very cumbersome due to the diverse structure of web pages. Most of the previous data extraction techniques were dealing with various data types such as text, audio, video and etc. but research works that are focusing on image as data are still lacking. Document Object Model (DOM) is an example of the state of the art of data extraction technique that is related to research work in mining image data. DOM was the method used to solve semi-structured data extraction from web. However, as the HTML documents start to grow larger, it has been found that the process of data extraction has been plagued with lengthy processing time and noisy information. In this research work, we propose an improved model namely Wrapper Extraction of Image using DOM and JSON (WEIDJ) in response to the promising results of mining in a higher volume of web data from a various types of image format and taking the consideration of web data extraction from deep web. To observe the efficiency of the proposed model, we compare the performance of data extraction by different level of page extraction with existing methods such as VIBS, MDR, DEPTA and VIDE. It has yielded the best results in Precision with 100, Recall with 97.93103 and F-measure with 98.9547.

Land Use Change Detection and Prediction Using Markov-CA and Publishing on the Web with Platform Map Server, Case Study: Qom Metropolis, Iran

To achieve sustainable development, detailed planning, control and management of land cover changes that occurs naturally or by human caused artificial factors, is essential. Urban managers and planners need a tool that represents them the information accurate, fast and in exact time. In this study, land use changes of 3 periods 1994-2002, 2002-2009, 2009-2015 and predictions of 2009, 2015 and 2023 were assessed. In this paper, Maximum Likelihood method was used to classify the images, so that after evaluation of accuracy, amount of overall accuracy for images of 2013 was 85.55 % and its Kappa coefficient was 80.03%. To predict land use changes, Markov-CA model was used that after assessing the accuracy, the amount of overall accuracy for 2009 was 82.57% and for 2015 was 93.865%. ThenWebGis application was designed via map server application and evoked shape files through map file and open layers to browser environment and for design of appearance of website Css, HTML and JavaScript languages were used. HTML is responsible for creating the foundation and overall structure of webpage but beautify and layout design is on Css.

Semantic Similarity between Web Documents Using Ontology

The World Wide Web is the source of information available in the structure of interlinked web pages. However, the procedure of extracting significant information with the assistance of search engine is incredibly critical. This is for the reason that web information is written mainly by using natural language, and further available to individual human. Several efforts have been made in semantic similarity computation between documents using words, concepts and concepts relationship but still the outcome available are not as per the user requirements. This paper proposes a novel technique for computation of semantic similarity between documents that not only takes concepts available in documents but also relationships that are available between the concepts. In our approach documents are being processed by making ontology of the documents using base ontology and a dictionary containing concepts records. Each such record is made up of the probable words which represents a given concept. Finally, document ontology’s are compared to find their semantic similarity by taking the relationships among concepts. Relevant concepts and relations between the concepts have been explored by capturing author and user intention. The proposed semantic analysis technique provides improved results as compared to the existing techniques.

Component Implementation of Gis-based IoT Trend Display and Control System

This article aims to design and implement GIS-based IoT trend display control system. Taking regional protection monitoring as the application scenario, this paper studies how to construct a system by using components and how to use GIS technology to organize the information collected by multiple sensors and visualize the information. It is proposed to build the IoT trend display and control system by using MVC framework based on B-S mode and the componentized web page structure by using React framework based on the MVVM pattern. The system will use API interfaces for JavaScript from the cloud GIS platform and a graphical display technology based on the HTML5 / CSS3 / JavaScript language to achieve visualization.

Responsive HTML generation using SuperSQL

PurposeWith the rapid spread of smartphones and tablets, it is becoming necessary for web developers to create responsive web pages which are visually appealing on devices of various sizes. However, building responsive UIs is a very challenging task, requiring deep knowledge of HTML and CSS. This paper aims to propose an approach to generate responsive web pages using SuperSQL, which is an extension of SQL that can format data retrieved from a database into various kinds of structured documents.Design/methodology/approachBy incorporating the methodology of bootstrap, a grid-based framework for front-end development, the authors have made it possible to create responsive web pages from simple SuperSQL queries. In addition, by utilizing SuperSQL’s unique feature that can describe the structure of the output web page, the authors have proposed and implemented a mechanism to automatically optimize the web content’s size and position.FindingsIn the evaluation, the authors created some actual web applications with and without the use of SuperSQL and compared the code amount (number of lines). As a result, when using the proposed system, the amount of code was reduced to about 1/5. The authors also compared the layout generated by the proposed automatic layout generation mechanism with the responsive layout that was generated manually. As a result, the automatic layout generation mechanism created the same layout as the manually created layout 74.8 per cent of the time, and the user satisfaction level turned out to be 85.8 per cent.Originality/valueThe way to generate a responsive HTML by using a single SuperSQL query, and the mechanism for automatic responsive layout generation.

Framework for location-aware search engine

Nowadays, a large part of the multimedia data on the Internet is generated with devices that automatically annotate them with location information However, free-form content on websites does not implicitly contain any geographical information. This is the biggest challenge for building a location-aware search engine. In this paper, we study how to extract location-aware information from the web. The key challenges are to detect location from a web page and to extract relevant information related to that location. We detect locations by identifying postal addresses using freely available gazetteers. Additional information for summarising the search results are titles and representative images, which we mine from the content using simple rule-based approaches utilising the structure of web pages. This information can be used to personalise search results for mobile users so that the results are relevant to their location.

Estimating similarity of rich internet pages using visual information

Traditional text-based web page similarity measures fail to handle rich-information-embedded modern web pages. Current approaches regard web pages as either DOM trees or images. However, the former only focuses on the web page structure, while the latter ignores the inner connections among different web page features. Therefore, they are not suitable for modern web pages. Hence, the idea of a block tree is introduced, which contains both structural and visual information of web pages. A visual similarity metric is proposed as the edit distance between two block trees. Finally, an experiment is undertaken, by cross-comparing 500 web pages, illustrating that the model appears to be highly accurate, empirically demonstrating that the metric is highly promising.

Improved Parallel PageRank Algorithm for Spam Filtering

Background/Objectives: PageRanking algorithm is a well known link based technique given by Google for indexing of its web pages. This algorithm works on the linking structure of web pages id est inbound and outbound links of pages. The existing Page Rank algorithm follows equal distribution law that is; it distributes the Page Rank of a web page evenly among all the outgoing links. The problem with the uniform distribution of Page Rank is that sometimes uninteresting pages got high Page Rank values. Methods/Statistical Analysis: This paper proposed an improved parallel Page Rank algorithm that un-uniformly distributes the Page Rank values among all the outgoing links. The proposed work has been implemented on NVIDIA Quadro 2000 GPU architecture using CUDA programming language. Findings: The proposed algorithm mitigates spam and provides better results in terms of computational time as compared to Parallel Page Rank, because it assigns higher priority to important pages and less priority to less important web pages. By assigning values in such a fashion important pages show an increase in the Page Rank value and unrelated pages that is spam pages show a decrease in Page Rank value. Application: The proposed work performs spam filtering by classifying important as well as irrelevant web pages.

Intent-Based Extensible Real-Time PHP Supervision Framework

Protecting Web applications is increasingly important due to their high popularity and wide adoption. Therefore, a multitude of protection techniques emerged in effort to secure Web applications, specifically considering valuable and private data commonly processed by such applications. Based on an overview of currently existing protection techniques, a generic and extensible PHP-oriented protection framework is proposed. The concept of application developer intent is introduced and compared with other concepts such as enforced security policies commonly used in existing protection approaches. The proposed framework is mainly focused on application developer intention understanding. Supervising the application execution in real-time makes it possible to detect deviations from the intended behavior and prevent potentially malicious activity. The additional aspects of application behavior, such as database-related communications or generated Web page structure, can be analyzed due to the extensible architecture of the framework.

Crowdsourcing Human Annotation on Web Page Structure

Parsing the semantic structure of a web page is a key component of web information extraction. Successful extraction algorithms usually require large-scale training and evaluation datasets, which are difficult to acquire. Recently, crowdsourcing has proven to be an effective method of collecting large-scale training data in domains that do not require much domain knowledge. For more complex domains, researchers have proposed sophisticated quality control mechanisms to replicate tasks in parallel or sequential ways and then aggregate responses from multiple workers. Conventional annotation integration methods often put more trust in the workers with high historical performance; thus, they are called performance-based methods. Recently, Rzeszotarski and Kittur have demonstrated that behavioral features are also highly correlated with annotation quality in several crowdsourcing applications. In this article, we present a new crowdsourcing system, called Wernicke, to provide annotations for web information extraction. Wernicke collects a wide set of behavioral features and, based on these features, predicts annotation quality for a challenging task domain: annotating web page structure. We evaluate the effectiveness of quality control using behavioral features through a case study where 32 workers annotate 200 Q&A web pages from five popular websites. In doing so, we discover several things: (1) Many behavioral features are significant predictors for crowdsourcing quality. (2) The behavioral-feature-based method outperforms performance-based methods in recall prediction, while performing equally with precision prediction. In addition, using behavioral features is less vulnerable to the cold-start problem, and the corresponding prediction model is more generalizable for predicting recall than precision for cross-website quality analysis. (3) One can effectively combine workers’ behavioral information and historical performance information to further reduce prediction errors.

A set of features to detect web security threats

The increasing growth of malicious websites and systems for distributing malware through websites is making it urgent the adoption of effective techniques for timely detection of web security threats. Current mechanisms may exhibit some limitations, mainly concerning the amount of resources required, and a low true positives rate for zero-day attacks. With this paper, we propose and validate a set of features extracted from the content and the structure of webpages, which could be used as indicators of web security threats. The features are used for building a predictor, based on five machine learning algorithms, which is applied to classify unknown web applications. The experimentation demonstrated that the proposed set of features is able to correctly classify malicious web sites with a high level of precision, corresponding to 0.84 in the best case, and recall corresponding to 0.89 in the best case. The classifiers reveal to be successful also with zero day attacks.

Fona

The Web 2.0 brought new requirements to the architecture of web systems. Web applications’ interfaces are becoming more and more interactive. However, these changes are severely impacting how disabled users interact through assistive technologies with the web. In order to deploy an accessible web application, developers can use WAI-ARIA to design an accessible web application, which manually implements focus and keyboard navigation mechanisms. This article presents a quantitative metric, named Fona, which measures how the Focus Navigation WAI-ARIA requirement has been implemented on the web. Fona counts JavaScript mouse event listeners, HTML elements with role attributes, and TabIndex attributes in the DOM structure of webpages. Fona’s evaluation approach provides a narrow analysis of one single accessibility requirement. But it enables monitoring this accessibility requirement in a large number of webpages. This monitoring activity might be used to give insights about how Focus Navigation and ARIA requirements have been considered by web development teams. Fona is validated comparing the results of a set of WAI-ARIA conformant implementations and a set of webpages formed by Alexa’s 349 top most popular websites. The analysis of Fona’s value for Alexa’s websites highlights that many websites still lack the implementation of Focus Navigation through their JavaScript interactive content.