Web Data Extraction Research Articles

Optimizing Data Extraction using Preprocessing for Enhanced Efficiency

With the exponential growth of the internet, an abundance of information has become readily available. Extracting valuable data from the web is crucial for applications such as meta-querying and comparison shopping. However, the heterogeneous nature of web information poses a significant challenge to the extraction process. The web can be classified into the surface or visible web and the deep or invisible web. While conventional search engines can index the surface web, they fall short when it comes to the deep web. To access the deep web, users must submit queries to web databases, and the results are encapsulated in dynamically generated web pages containing data records. Traditional search engines struggle to index these dynamic pages, necessitating a specialized program for efficient information extraction from the deep web. Web search engines generate result pages based on user queries, making it crucial to automatically extract data from these pages for various applications. In this context, we propose an innovative data extraction method called Effective Data Extraction using Preprocessing (EDEP). The EDEP approach begins by parsing the input HTML page, constructing a tag tree, and subsequently eliminating irrelevant tags from the tree. Notably, our system efficiently handles scenarios where auxiliary information, such as recommendations or comments, is intermixed between query result records (QRRs), causing them to be non-contiguous. EDEP also effectively manages result pages containing single QRRs. Through experimental results, it is evident that EDEP outperforms existing data extraction methods, showcasing its efficacy in handling the complexities associated with web data extraction.

The Making of an Data Pipeline

This paper details the development and implementation of a data engineering pipeline designed for the extraction, transformation, and loading (ETL) of data from a web-based directory. The project involves using asynchronous web scraping techniques to gather user details from a local business directory, transforming the data into a structured format, and loading it into a storage solution. The pipeline utilises Python, the HTTPX library for asynchronous HTTP requests, BeautifulSoup for HTML parsing, and Amazon S3 for data storage. By leveraging these technologies, the pipeline demonstrates an efficient approach to handling large-scale web data extraction and processing, significantly reducing the time required to gather and organise data from multiple web pages. This paper provides insights into the architecture, implementation, and performance of the ETL pipeline, highlighting the benefits and challenges of using asynchronous programming in data engineering.

Optimizing Data Extraction using Preprocessing for Enhanced Efficiency

With the exponential growth of the internet, an abundance of information has become readily available. Extracting valuable data from the web is crucial for applications such as meta-querying and comparison shopping. However, the heterogeneous nature of web information poses a significant challenge to the extraction process. The web can be classified into the surface or visible web and the deep or invisible web. While conventional search engines can index the surface web, they fall short when it comes to the deep web.
 To access the deep web, users must submit queries to web databases, and the results are encapsulated in dynamically generated web pages containing data records. Traditional search engines struggle to index these dynamic pages, necessitating a specialized program for efficient information extraction from the deep web. Web search engines generate result pages based on user queries, making it crucial to automatically extract data from these pages for various applications.
 In this context, we propose an innovative data extraction method called Effective Data Extraction using Preprocessing (EDEP). The EDEP approach begins by parsing the input HTML page, constructing a tag tree, and subsequently eliminating irrelevant tags from the tree. Notably, our system efficiently handles scenarios where auxiliary information, such as recommendations or comments, is intermixed between query result records (QRRs), causing them to be non-contiguous. EDEP also effectively manages result pages containing single QRRs.
 Through experimental results, it is evident that EDEP outperforms existing data extraction methods, showcasing its efficacy in handling the complexities associated with web data extraction.

Understanding the relationships between hotel variables in Almaty, Kazakhstan: an investigation using booking.com data

Object: The purpose of this article is to study relationships between hotel variables such as star ratings, prices, customer ratings, distance from the city center, and review numbers for hotels in Almaty, Kazakhstan.Methods: Data for this study was collected quantitatively from Booking.com using the web scraping tool Parsehub, a popular tool in contemporary research for web data extraction. Following data collection and the removal of missing data, 69 cases out of 96 were retained. To address the research questions, the study uses descriptive statistics to analyze data by star rating categories, performs One Way ANOVA test to determine differences between variables, and conducts correlation analysis. The analysis tools used include Jamovi and SPSS 23.Findings: The findings indicate significant differences in customer ratings based on star ratings, a positive correlation between star ratings and prices, a weak, negative relationship between customer ratings and distance from the city center, and a weak positive correlation between review numbers and customer ratings.Conclusions: These insights contribute to a better understanding of the hotel industry in Almaty. Results demonstrate the importance of star ratings, customer reviews, price, and location in understanding the dynamics of the hotel market.

PEMODELAN TOPIK TERKAIT BANJIR PADA TWITTER DENGAN MENGGUNAKAN LATENT DIRICHLET ALLOCATION

In this background discusses the topic of tweet about Flooding on Twitter using the keyword "Flood". Tweet data was taken from June 1, 2021 to June 2, 2021 with the number of tweet data obtained, which was 2000 tweets. The number of tweets related to flooding has not been analyzed so that the topics contained in it are not yet known. Research . Modeling topics related to floods in Indonesia on Twitter social media with the LDA method. Research. This study uses experimental methods with several variables to test hypotheses. Then the data is processed with stages, namely web data extraction, preprocessing, feature extraction, topic modeling using latent dirichlet allocation algorithms, visualization, and analysis. Research. The results of the topic coherence stage were carried out a search for the most optimal topic from the 20 topics that had been determined at the beginning. The results of topic coherence for 20 topics concluded that for topic 10 it has a total topic value of 0.41 and has an ideal topic modeling result and is in accordance with the provisions. Conclusion : Based on the results of the discussion of topic coherence, it can be concluded that the most ideal number of topics is topic 10 because it has the highest value compared to other topics. The advice here is to be able to display or get flood information in Indonesia in real time and accurately.

0934 An Analysis of 5,212 Sleep Medicine Providers’ Online Patient Reviews

Abstract Introduction Online provider rating websites are an increasingly popular medium for patients to provide reviews & feedback and obtain information about varying healthcare providers. There have yet to be studies on what factors significantly impact sleep medicine provider ratings. Healthgrades is a popular healthcare provider rating database with over 11 million patient reviews. This study analyzes various dimensions to determine what factors correlate with average sleep medicine provider ratings to improve patient satisfaction. Methods Data were collected on 5,212 sleep medicine providers’ online profiles using web data extraction software. Factors such as medical degree, gender, age, and ethnicity were compared to online ratings using Pearson correlation and ANOVA analysis via statistics software. Results Variables with positive correlation in provider rating include the presence of a headshot (pc=0.106, p< 0.01), designation of trustworthiness (pc=0.152, p< 0.01), and presence of care philosophy statement (pc=0.105, p< 0.01). The only variable with a negative correlation in provider rating was increasing age (p=-0.109, p< 0.01). A one-way ANOVA revealed that there was a statistically significant difference in mean rating between different healthcare degrees (F(122.271, 5062.570)=[17.955], p=<.001). Tukey’s HSD Test for multiple comparisons found that the mean value of ratings was significantly different between allopathic physicians and sleep medicine dentists (p=0.000, 95% C.I.=[-0.943,0.070]), physician assistants (p=0.000, 95% C.I.=[-1.268,-0.239]), and nurse practitioners (p=0.000, 95% C.I.=[-0.941,-0.188]). There was no statistically significant difference in mean online rating between allopathic physicians and osteopathic physicians (p=0.505), sleep psychologists (p=0.344), or foreign medical graduates (p=0.998), respectively. Conclusion This study suggests patients perceive sleep medicine providers’ online profiles with headshots, care philosophy statements, and designation of trustworthiness as contributing factors when rating. Additionally, degree type and age matter when rating online. Patients do not value factors such as ethnicity or gender strongly when rating their sleep medicine provider. Further study is needed to determine factors influencing non-physician providers’ higher ratings when compared to U.S. medical graduates. This study demonstrates the importance of building an online presence for sleep medicine practices and identifying characteristics patients value. The expectation is that this information can be used to improve the quality of care delivered in the field of sleep medicine. Support (if any)

An efficient regular expression inference approach for relevant image extraction

Traditional approaches for extracting relevant images automatically from web pages are error-prone and time-consuming. To improve this task, operations such as preparing a larger dataset and finding new features are used in the web data extraction approaches. However, these operations are difficult and laborious. In this study, we propose a fully-automated approach based on alignment of regular expressions to automatically extract the relevant images from web pages. The automatically constructed regular expressions has been applied to a classification task for the first time. In this respect, a multi-stage inference approach is developed for generating regular expressions from the attribute values of relevant and irrelevant image elements in web pages. The proposed approach reduces the complexity of the alignment of two regular expressions by applying a constraint on a version of the Levenshtein distance algorithm. The classification accuracy of regular expression approaches is compared with the naive Bayes, logistic regression, J48, and multilayer perceptron classifiers on a balanced relevant image retrieval dataset consisting of 360 image element samples for 10 shopping websites. According to the cross-validation results, the regular expression inference-based classification achieved a 0.98 f-measure with only 5 frequent n-grams, and it outperformed other classifiers on the same set of features. The classification efficiency of the proposed approach is measured at 0.108 ms, which is very competitive with other classifiers.

Method of determining the attribution of printed documents

The rapid development of modern information technologies encourages organizations and enterprises to introduce innovative technologies in the information provision of their activities. Currently, this has a great impact on the successful implementation of organizational management and the adoption of correct strategic decisions. Effective processing of ever-growing amounts of information, the main part of which consists of documents and reports of various formats, is possible only under the condition of automated verification and processing. Information search in unstructured text is very difficult, because it contains a large amount of information, which requires the use of specific processing methods and algorithms to obtain useful knowledge. The article summarizes the results of an experimental study of the method of determining the attribution of an electronic document. The role of information processing and the identification of models and trends in it, which help to make decisions, as well as the principles of intelligent data analysis, are analyzed. Areas of intellectual text analysis, such as data collection, web data processing, information search and extraction, computer linguistics and natural language processing, are singled out. The expediency of implementing a prototype software product that reflects the operation of the method and shows that the method works quickly and stably has been proven.

SFTM: Fast matching of web pages using Similarity-based Flexible Tree Matching

Tree matching techniques have been investigated in many fields, including web data mining and extraction, as a key component to analyze the content of web pages. However, when applied to existing web pages, traditional tree matching approaches, covered by algorithms like Tree-Edit Distance (TED) or XyDiff, either fail to scale beyond a few hundred nodes or exhibit a relatively low accuracy.In this article, we therefore propose a novel algorithm, named Similarity-based Flexible Tree Matching (SFTM), which enables high accuracy tree matching on real-life web pages, with practical computation times. We approach tree matching as an optimization problem and leverage node labels and local topology similarity in order to avoid any combinatorial explosion. Our practical evaluation demonstrates that SFTM significantly improves the state of the art in terms of accuracy, while allowing computation times significantly lower than the most accurate solutions. By gaining on these two dimensions, SFTM therefore offers an affordable solution to match complex trees in practice.

Surviving in financial advice deserts: limited access to financial advice and retirement planning behavior

PurposeThis study introduces the concept of financial advice deserts (FADs), including financial advice received from personal financial advisors (PFAs) and Certified Financial Planners™ (CFP professionals) and investigates the association between living in these FAD states and the retirement planning activities of individuals.Design/methodology/approachThis study uses merged data gathered from multiple sources including (1) available state-level information on CFP professionals from the CFP board website, (2) state-level information on PFAs from the US Bureau of Labor Statistics and (3) individual levels of retirement planning behavior and other personal characteristics from the 2018 FINRA National Financial Capability Study. Using web data extraction tools and logistic regression analyses, this study examines the association between a series of individual retirement planning activities and living in the FAD states.FindingsThe study found that living in the FAD states was negatively associated with both having retirement accounts and contributing regularly to retirement accounts. Overall, the findings of this study underscore the need for providing greater access to financial advice and improving financial literacy among financially marginalized populations who are residing in FAD states in the United States of America.Originality/valueThis study makes unique contributions to the literature by raising the issue of geographic inequality in terms of access to financial advice and introducing the innovative notion of FADs. The findings provide fresh insights into the understanding of retirement planning and preparedness from the perspective of state-level inequality of financial advice through PFAs and CFP professionals, thereby expanding the previous knowledge that emphasizes only individual- and household-level differences. Significant implications for public policies and practitioners are also discussed.

Bot crawler to retrieve data from Facebook based on the selection of posts and the extraction of user profiles

Introduction— Data can currently be found within organizations and outside of them, they are growing exponentially. Today, the information available on the Internet and social networks has become a generator of value, through the effective analysis of a specific situation, using techniques and methodologies with which content-based solutions can be proposed, and thus achieve, execute timely, intelligent and assertive decision-making processes. Objective— The main objective of this work is to development of a Bot Crawler, which allows extracting information from Facebook without access restrictions, or request for credentials, based on web crawling and scraping techniques, through the selection of HTML tags, to track and be able to define patterns. Methodology— The development of this project consisted of four main stages: A) Teamwork with SCRUM, B) Comparison of web data extraction techniques, C) Extraction and validation of permissions to access the data in Facebook, D) Development of the bor crawler. Results— As a result of this process, a graphical interface was created to review the process of obtaining data derived from user profiles in this social network. Conclusions— As a result of this process, a graphical interface is created that allows checking the process of obtaining data derived from user profiles of this social network.

Aesthetic Trends and Semantic Web Adoption of Media Outlets Identified through Automated Archival Data Extraction

The last decade has been a time of great progress in the World Wide Web and this progress has manifested in multiple ways, including both the diffusion and expansion of Semantic Web technologies and the advancement of the aesthetics and usability of Web user interfaces. Online media outlets have often been popular Web destinations and so they are expected to be at the forefront of innovation, both in terms of the integration of new technologies and in terms of the evolution of their interfaces. In this study, various Web data extraction techniques were employed to collect current and archival data from news websites that are popular in Greece, in order to monitor and record their progress through time. This collected information, which took the form of a website’s source code and an impression of their homepage in different time instances of the last decade, has been used to identify trends concerning Semantic Web integration, DOM structure complexity, number of graphics, color usage, and more. The identified trends were analyzed and discussed with the purpose of gaining a better understanding of the ever-changing presence of the media industry on the Web. The study concluded that the introduction of Semantic Web technologies in online media outlets was rapid and extensive and that website structural and visual complexity presented a steady and significant positive trend, accompanied by increased adherence to color harmony.

Excise taxes and pricing activities of e-liquid products sold in online vape shops

BackgroundAlthough e-cigarette excise taxes have great potential to prevent the initiation and escalation of e-cigarette use, little information is available on pricing activities of online vape shops, and how well...

Research on the Key Technology of Web Data Extraction and Mining Based on the Probability Distribution

In the development of internet technology innovation, the quantity of big data information continues to rise, showing the characteristics of dynamic, heterogeneous, massive, and so on. How to explore valuable and potential knowledge from the network system is the main topic of research and scholars. Based on the understanding of web data extraction and web mining technology, this paper makes a systematic study of web text mining, proposes the most commonly used hidden Markov model in probability distribution, and constructs the corresponding text mining method. The hidden Markov model is a statistical model, which is mainly used to represent a Markov process with unknown parameters. The difficulty in practical application lies in how to define the hidden parameters of the process from observable parameters and then use these parameters to conduct in-depth research. Finally, the practical results show that the hidden Markov model can not only obtain the information of different regions but also deeply analyze the data set, which proves the feasibility of this research technology.

Investigating the Country of Origin and the Role of the .eu TLD in External Trade of European Union Member States

The Internet, and specifically the World Wide Web, has always been a useful tool in the effort to achieve more outward-looking economies. The launch of the .eu TLD (top-level domain) in December of 2005 introduced the concept of a pan-European Internet identity that aimed to enhance the status of European citizens and businesses on the global Web. In this study, the countries of origin of websites that choose to use the .eu TLD are investigated and the reasoning behind that choice, as well as its relation to each country’s economy and external trade are discussed. Using the Web as a tool, information regarding a vast number of existing .eu websites was collected, through means of Web data extraction, and this information was analyzed and processed by a detailed algorithm that produced results concerning each website’s most probable country of origin based on a multitude of factors. This acquired knowledge was then used to investigate relations with each member-state’s presence in its local ccTLD, its GDP and its external trade revenue. The study establishes a correlation between presence in the .eu TLD and external trade that is both independent of a country’s GDP and stronger than the relation between its local ccTLD presence and external trade.

How to regulate vaping products to advance cancer prevention: Evidence from online stores.

e22502 Background: The use of electronic cigarettes has significantly increased in the past decade, and online stores are one of the most common outlets for e-cigarette users to purchase vaping products. The goal of this study is to examine the appeal of different e-liquid flavors to vapers, using a revealed preference method to estimate consumer demand. Methods: We collected data on e-liquid sales prices, volume (in ml), and a variety of product flavors, from online vaping stores using web data extraction. We standardized prices for e-liquid products and used the hedonic pricing model to assess the associations between standardized e-liquid prices and flavors, controlling for nicotine concentration (in mg) and product VG/PG ratio. In total, we collected data of 10,870 e-liquid products from four popular online vaping stores that sell nationwide across the U.S. Results: Higher e-liquid prices (standardized to $/ml) are associated with higher nicotine concentrations. E-liquid products with fruit tobacco flavors are associated with higher sales prices. E-liquid prices are higher when VG/PG ratio deviates from 70/30 except for 90/10: the prices of e-liquid products with a 90/10 VG/PG ratio are lower than the prices of products with a 70/30 ratio. All of the findings above are statistically significant at a 5% level. Conclusions: Higher nicotine concentration and fruit flavors are associated with higher e-liquid prices in national online vaping stores, suggesting that they are preferred attributes and are in greater demand among vapers in the market of heterogeneous e-liquid products.

Web Scraping and Regression Analysis based on Machine Learning for COVID-19 with Rapid Software Platform

Since the recent incidence of global COVID-19 pandemic, expertise from different domains including scientists, clinicians, and healthcare experts keep on exploring for technologies to manage the COVID-19 data. Updated and accurate data collection is very critical for them to make a more effective and efficient decision on any aspects of the emergency consequences and events. Although some of them are inexpert data scientists, the important skills and knowledges to extract the recent data on COVID-19 is web data extraction and analysis. While tremendous of literature can be referred from the academic databases, it is difficult to find the report that presents the basis and fundamental methods for implementing web data analysis in a simple way with a rapid software platform. This paper demonstrates a simple framework for implementing web data extraction or web scraping to be analyzed in a rapid software platform. Python scripting language is the simple tool to conduct the web scraping method while RapidMiner is the rapid software for implementing the data visualization and analysis. Simple linear regression based on machine learning approach has been implemented with the RapidMiner to predict COVID-19 death based on the collected data. This paper will be useful for academicians and industry practitioners to conduct a more robust data analysis to accommodate a more challenge issue such as big data analytics in any domains.

An Efficient Mechanism for Deep Web Data Extraction Based on Tree-Structured Web Pattern Matching

The World Wide Web comprises of huge web databases where the data are searched using web query interface. Generally, the World Wide Web maintains a set of databases to store several data records. The distinct data records are extracted by the web query interface as per the user requests. The information maintained in the web database is hidden and retrieves deep web content even in dynamic script pages. In recent days, a web page offers a huge amount of structured data and is in need of various web-related latest applications. The challenge lies in extracting complicated structured data from deep web pages. Deep web contents are generally accessed by the web queries, but extracting the structured data from the web database is a complex problem. Moreover, making use of such retrieved information in combined structures needs significant efforts. No further techniques are established to address the complexity in data extraction of deep web data from various web pages. Despite the fact that several ways for deep web data extraction are offered, very few research address template-related issues at the page level. For effective web data extraction with a large number of online pages, a unique representation of page generation using tree-based pattern matches (TBPM) is proposed. The performance of the proposed technique TBPM is compared to that of existing techniques in terms of relativity, precision, recall, and time consumption. The performance metrics such as high relativity is about 17-26% are achieved when compared to FiVaTech approach.

Building Self-Healing Feature Based on Faster R-CNN Deep Learning Technique in Web Data Extraction Systems

Web data extraction has evolved over the years with extracting data from documents to today’s World Wide Web (WWW). The WWW growth has placed data at the centre of this ecosystem and benefited society at large, businesses and consumers. The proposed system uses deep learning technique, Faster region convolutional neural network (R-CNN) for automated navigation, extraction of data and self-healing of data extraction engine to adapt to dynamic changes in website layout. The proposed system trains the Faster R-CNN model for detection of product in the web page using bounding box image detection technique and extracts product details with high extraction accuracy. Deep learning technique has advanced rapidly in the different fields for image detection, but its application in data extraction makes this paper unique. An ecommerce retail website is used as real-world example to prove the self-healing capability of the proposed automated web data extraction system.

Trends in web data extraction using machine learning

Web data extraction has seen significant development in the last decade since its inception in the early nineties. It has evolved from a simple manual way of extracting data from web page and documents to automated extraction to an intelligent extraction using machine learning algorithms, tools and techniques. Data extraction is one of the key components of end-to-end life cycle in web data extraction process that includes navigation, extraction, data enrichment and visualization. This paper presents the journey of web data extraction over the last many years highlighting evolution of tools, techniques, frameworks and algorithms for building intelligent web data extraction systems. The paper also throws light into challenges, opportunities for future research and emerging trends over the years in web data extraction with specific focus on machine learning techniques. Both traditional and machine learning approaches to manual and automated web data extraction are experimented and results published with few use cases demonstrating the challenges in web data extraction in the event of changes in the website layout. This paper introduces novel ideas such as self-healing capability in web data extraction and proactive error detection in the event of changes in website layout as an area of future research. This unique perspective will help readers to get deeper insights in to the present and future of web data extraction.