Web Scraper Research Articles

Lessons from web scraping coroners' Prevention of Future Deaths reports.

In England and Wales, coroners are required to write Prevention of Future Deaths reports when a death is deemed preventable so that action is taken to avert similar deaths. Since July 2013, Prevention of Future Deaths reports have been openly available via the Courts and Tribunals Judiciary website (https://www.judiciary.uk/prevention-of-future-death-reports/). However, their presentation to date have been insufficient to identify trends and learn lessons. We designed a web scraper to create the Preventable Deaths Tracker (https://preventabledeathstracker.net/). On 22 June 2022, 4001 PFDs were scraped, analysed, and compared to the Office of National Statistics' preventable mortality statistics. This commentary summarises the key findings and offers recommendations to improve the Prevention of Future Deaths system so lessons can be learnt to avert preventable deaths.

Exploring the Key Design Attributes of Cafés from User-Generated Contents - A Case Study of Two Cafés of Unique Interior Designs in Yeonnam-dong, Seoul -

In the last couple of decades, the use of social media has consistently increased. Different social media platforms allow users to interact and create diverse types of content, making them able to share their opinions. With brands getting involved in these platforms, it is important to know how their image is portrayed by their customers on social media. This includes the café industry. The purpose of our study is twofold. One is to understand which design attributes make a café appealing to social media users, and another is to examine whether the Google Cloud Vision API is able to differentiate the key design attributes using user-generated content, which are Instagram posts in this research. In order to achieve these aims, we conducted a comparative case study using two cafés of unique interior designs located in Yeonnam-dong in Seoul, South Korea. Specifically, Greem Cafe and Perception Coffee were used. Using these cases as hashtags, we scraped the posts using a Python web scraper and screened out the posts irrelevant to the cafés. After, we ran them through the Google Cloud Vision API to obtain the labels and screened out the labels irrelevant to the interior designs, such as people and amenities. Finally, we were able to categorize the label to different design attributes and compare and contrast the labels from two cafés. The main differences shown in the results from the labels are that Greem Café had Color and Cartoon attribute labels like “Black-and-White” and “Drawing” that clearly represent the cartoonish interior design style. On the other hand, Perception Coffee’s most frequent label, “Wood” along with other labels like “Aeolian Landform” identify their wooden ceiling design. With these results, it is shown that the Google Cloud Vision API is able to distinguish the main design elements from both cafés. This research utilizes a new tool that can be useful for future researchers and designers that deal with big data, also this research brings insight for designers at the time of creating new places by analyzing data made by consumers and promotes the usage of social media data as a tool for design thinking.

Empirical Research on Data Science Talent Training and Enterprise Demand in Colleges and Universities

The advent of the information age has given rise to a huge amount of data, and it becomes crucial to use the data well and bring its value into play. How colleges and universities train data science talents to meet the huge demand of the industry is of great significance. In this paper, the demand information of data science talents from recruitment websites was crawled by a typical web scraper and then carried out pre-processing, statistical analysis, and visual display. Through fully excavating the job information and summarizing the law of enterprise demand, constructive suggestions were put forward for data science talent training in colleges and universities, providing a reference basis for talent training units to develop training programs.

SwaRegex: a lexical transducer for the morphological segmentation of swahili verbs

The morphological syntax of the Swahili verb comprises 10 slots. In this work, we present SwaRegex, a novel rule-based model for the morphological segmentation of Swahili verbs. This model is designed as a lexical transducer, which accepts a verb as an input string and outputs the morphological slot occupied by the morphemes in the input string. SwaRegex is based on regular expressions developed using the C# programming language. To test the model, we designed a web scraper that obtained verbs from an online Swahili dictionary. The scrapper separated the corpus into two datasets: dataset A, comprising 163 verbs Bantu origin; and dataset B, containing the entire set of 715 non-Arabic verb entries obtained by the web scrapper. The performance of the model was tested against a similar model designed using the Xerox Finite State Tools (XFST). The regular expressions used in both models were the same. SwaRegex outperformed the XFST model on both datasets, achieving a 98.77% accuracy on dataset A, better than the XFST model by 41.1%, and a 68.67% accuracy on dataset B, better than the XFST model by 38.46%. This work is beneficial to prospective learners of Swahili, by helping them understand the syntax of Swahili verbs, and is an integral teaching aid for Swahili. Search engines will benefit from the lexical transducer by leveraging its finite state network when lemmatizing search terms. This work will also create more opportunities for more research to be done on Swahili.

Implementation of marketplace data in the production of Consumer Price Index in Indonesia

Digital Economy in recent years, especially in Southeast Asia, including Indonesia, is growing rapidly. E-commerce is one part of the Digital Economy. BPS-Statistics Indonesia as a Non-ministerial Government Agency responsible directly to the president has conducted an E-commerce Survey in 2019. From this publication, it is concluded that the interest of Indonesian traders using the internet in selling in recent years has increased. So, the urgency of using e-commerce data in its application in official Statistics is increasingly needed. Several studies have carried out the application of e-commerce data in the calculation of The Consumer Price Index (CPI). In this research, e-commerce data is applied with a case study using the data from one of online marketplaces in Indonesia in calculating CPI at city level in Java. The purpose of this study is to compare the marketplace-based CPI data and BPS-Statistics’ survey-based CPI. The data is collected through web scraping techniques and followed by preprocessing data and analyzed descriptively. Web scraper that is built can be used in obtaining data. Commodity-level CPI with marketplace data tends to have relatively large prices which result in higher CPI being compared to BPS-Statistics CPI. Meanwhile, at the expenditure group level, the CPI between the two approaches is broadly similar in general.

Car Price Predictor

The prediction of price for a car has been more popular in research area, and it needs predominant effort and information about the experts of this particular field. The number of different attributes is measured and also it has been considerable to predict the result in more reliable and accurate. To find the price of used vehicles a well defined model has been developed with the help of three machine learning techniques such as Artificial Neural Network, Support Vector Machine and Random Forest. These techniques were used not on the individual items but for the whole group of data items. This data group has been taken from some web portal and that same has been used for the prediction. The data must be collected using web scraper that was written in PHP programming language. Distinct machine learning algorithms of varying performances had been compared to get the best result of the given data set. The final prediction model was integrated into Java application.

Recognition of Culturally Diverse Cuisine in Popular Nutrition Apps

ObjectivesNutrition apps have been designed to assist users in determining the nutritional value of their food in alliance with their dietary restrictions and/or health. However, many of these apps do not incorporate or recognize the cuisine of the diverse communities that they serve. In this work we assessed several of the most popular free apps and compared with the most popular dishes in Latin America, Europe, India, East Asia, West Africa, and the American South to determine the rate of recognition of culturally diverse cuisines in popular nutrition apps.MethodsWe built a consensus database of over 150 culturally popular foods via a web scraper for foods from Latin American, Europe, India, East Asia, West Africa, and the American South. Researched & logged popular dishes from International cuisines with their corresponding nutritional data. Each food item was run against Calorie Mama, Lose It, My Fitness Pal and My Plate to determine the recognition and availability of the nutritional value.ResultsEuropean and Indian dishes were better recognized (greater than 70%) across all apps than West African and Latin American dishes. Of the dishes that were recognizable, most were due to manual entry and lacked some of the nutritional information, making the apps less useful.ConclusionsWe note a bias among popular nutrition apps in not taking into account cuisines, from both international and regional American communities, leaving many communities underserved and contributing to health inequalities gap.Funding SourcesN/A.

Predicting Policy: A Psycholinguistic Artificial Intelligence in the United Nations

In organizational theory, institutionalists generally make predictions of corresponding context and policy outcome based on structural processes. Psychoanalytic theory, in contrast, focuses on the rhetorical framing rather than the environment of a policy for predictive outcomes. This study aims to explore the debate over policy prediction by developing a supervised machine learning model to predict for policy success and context in the United Nations (UN). Through data collected with a python web scraper on all UN meetings in the General Assembly (GA) and Security Council (SC) between 1994 and 2020, we parse motions, policies, and conflict indicators, before passing meeting records through the Linguistic Inquiry and Word Count (LIWC) psycholinguistic algorithm. Next, we build 12 different machine learning models to predict for policy passage and context using preprocessed motion and LIWC data; results demonstrate that the psychoanalytic models better predicted for both context and policy outcomes than the institutionalist models, suggesting that the classical political axiom, “actions speak louder than words,” may not be supported by the empirical evidence.

Risk Avoidance Behavior on Darknet Marketplaces

The current study employs a quasi-experimental design to test the efficacy of situational crime prevention (SCP) in understanding risk avoidance behavior in a darknet environment. Specifically, we deployed a web scraper to extract data from a popular darknet market. We then used these data to assess change in vendors and customers’ behavior following the deliverance of a warning message from the market administrator regarding the former market administrator’s intention to scam market participants. In the one-month following the message, vendors posted fewer advertisements and customers spent less and made fewer transactions on the marketplace. These findings expand the scope of SCP and provide evidence for Hutchings and Holt’s notion that darknet markets can be disrupted by means of gossip.

Gayrimenkul Fiyat Tahmini ve Alttaki Özelliklerin Analizi İçin C4.5 – CART Karar Ağacı Modeli

The machine learning approaches are used in different domains for price prediction. Real estate price prediction comes to fore in recent years. However, most of the studies focus on the prediction performance and the factors affecting the price are often ignored. In this study, a C4.5 – CART model to predict the residential real estate prices is developed. This model is capable of predicting both numeric and categorical price for real estate properties. In addition, the factors affecting the price are reveled and analyzed in detail. The performance of the developed model is compared to Direct Capitalization model, which is used as a gold standard in the domain. Both models are tested on a dataset that includes updated real time data that is gathered by a web scraper. For numeric prediction, RMSE of the developed model is 13.169 and 358.69 for the Direct Capitalization model. KAPPA and accuracy is used for the categorical prediction. The model has 81% KAPPA and 88% accuracy.

Social recruiting: an application of social network analysis for preselection of candidates

PurposeThe paper aims to studiy social recruiting for finding suitable candidates on social networks. The main goal is to develop a methodological approach that would enable preselection of candidates using social network analysis. The research focus is on the automated collection of data using the web scraping method. Based on the information collected from the users' profiles, three clusters of skills and interests are created: technical, empirical and education-based. The identified clusters enable the recruiter to effectively search for suitable candidates.Design/methodology/approachThis paper proposes a new methodological approach for the preselection of candidates based on social network analysis (SNA). The defined methodological approach includes the following phases: Social network selection according to the defined preselection goals; Automatic data collection from the selected social network using the web scraping method; Filtering, processing and statistical analysis of data. Data analysis to identify relevant information for the preselection of candidates using attributes clustering and SNA. Preselection of candidates is based on the information obtained.FindingsIt is possible to contribute to candidate preselection in the recruiting process by identifying key categories of skills and interests of candidates. Using a defined methodological approach allows recruiters to identify candidates who possess the skills and interests defined by the search. A defined method automates the verification of the existence, or absence, of a particular category of skills or interests on the profiles of the potential candidates. The primary intention is reflected in the screening and filtering of the skills and interests of potential candidates, which contributes to a more effective preselection process.Research limitations/implicationsA small sample of the participants is present in the preliminary evaluation. A manual revision of the collected skills and interests is conducted. The recruiters should have basic knowledge of the SNA methodology in order to understand its application in the described method. The reliability of the collected data is assessed, because users provide data themselves when filling out their social network profiles.Practical implicationsThe presented method could be applied on different social networks, such as GitHub or AngelList for clustering profile skills. For a different social network, only the web scraping instructions would change. This method is composed of mutually independent steps. This means that each step can be implemented differently, without changing the whole process. The results of a pilot project evaluation indicate that the HR experts are interested in the proposed method and that they would be willing to include it in their practice.Social implicationsThe social implication should be the determination of relevant skills and interests during the preselection phase of candidates in the process of social recruitment.Originality/valueIn contrast to previous studies that were discussed in the paper, this paper defines a method for automatic data collection using the web scraper tool. The described method allows the collection of more data in a shorter period. Additionally, it reduces the cost of creating an initial data set by removing the cost of hiring interviewers, questioners and people who collect data from social networks. A completely automated process of data collection from a particular social network stands out from this model from currently available solutions. Considering the method of data collection implemented in this paper, the proposed method provides opportunities to extend the scope of collected data to implicit data, which is not possible using the tools presented in other papers.

An analysis of public opinions regarding Internet-famous food: a 2016–2019 case study on Dianping

PurposeInternet-famous food (also known as “online celebrity” food) is very popular in the digital age. This study aims to investigate consumer attitudes and understand consumer behavior towards Internet-famous food.Design/methodology/approachThe authors collected 136,835 online comments regarding “Internet-famous food” from Dianping platform between 2016 and 2019 using a web scraper. A sentiment lexicon for Internet-famous food was constructed, and sentiment analysis is further conducted to understand consumer attitudes. Additionally, the authors use topic analysis and time series analysis to study consumer behavior.FindingsSentiment analysis showed that the number of consumers' comments decreased over time with the attitudes being overall positive, and the Internet-famous food industry has a positive prospect; time series analysis showed that the consumption of Internet-famous food was not affected by the season; topic analysis showed that consumers' comments on Internet-famous food were rich with a large variety, covering food categories, brand, quality, service, environment and price.Originality/valueTo the authors’ knowledge, limited research has focused on public opinions regarding “Internet-famous food”. This is the first study on consumer behavior towards Internet-famous food. This article provides a unique insight into the purchasing behavior and attitude of Chinese Internet-famous food consumers through text mining.

Consumer Engagement With Visual Content on Instagram

Despite Instagram’s popularity among the Y and Z generation consumers, social media research investigating factors that can influence customer engagement on Instagram remains limited. The uses and gratifications theory underpins this study which investigates how social media marketing of prominent Malaysian brands on Instagram impacts customer engagement. ScrapeStorm web scraper tool was used to extract posts and comments from the public Instagram accounts of four notable Malaysian brands. The findings reveal that different features of social media, affect consumer engagement differently. Social media influencers had a positive impact on consumption and creation-based customer engagement. Contextual features are negatively correlated with contribution-based customer engagement. In terms of content, videos had greater engagement rates, followed by images of people and images without people. Results also show that consumers reacted positively to informative textual content. Some implications for theory and practice are highlighted.

Infant Safe Sleep Practices as Portrayed on Instagram: Observational Study.

BackgroundParenting practices are highly influenced by perceived social norms. Social norms and American Academy of Pediatrics (AAP) guidelines for infant safe sleep practices are often inconsistent. Instagram has become one of the most popular social media websites among young adults (including many expectant and new parents). We hypothesized that the majority of Instagram images of infant sleep and sleep environments are inconsistent with AAP guidelines, and that the number of “likes” for each image would not correlate with adherence of the image to these guidelines.ObjectiveThe objective of this study was to determine the extent of adherence of Instagram images of infant sleep and sleep environments to safe infant sleep guidelines.MethodsWe searched Instagram using hashtags that were relevant to infant sleeping practices and environments. We then used an open-source web scraper to collect images and the number of “likes” for each image from 27 hashtags. Images were analyzed for adherence to AAP safe sleep guidelines.ResultsA total of 1563 images (1134 of sleeping infant; 429 of infant sleep environment without sleeping infant) met inclusion criteria and were analyzed. Only 117 (7.49%) of the 1563 images were consistent with AAP guidelines. The most common reasons for inconsistency with AAP guidelines were presence of bedding (1173/1563, 75.05%) and nonrecommended sleep position (479/1134, 42.24%). The number of “likes” was not correlated with adherence of the image to AAP guidelines.ConclusionsAlthough individuals who use Instagram and post pictures of sleeping infants or infant sleep environments may not actually use these practices regularly, the consistent portrayal of images inconsistent with AAP guidelines reinforces that these practices are normative and may influence the practice of young parents.

Prevalence of missing nutrition label and ingredients list information on e‐shops of major chain supermarkets in Hong Kong

AbstractDue to the coronavirus pandemic, online food purchases have increased. Nutrition labels and ingredients lists are important to assist individuals in making healthy food choices, but their provision is not mandatory for online stores in Hong Kong. This study aimed to investigate the prevalence of missing nutrition labels and ingredients list information on the e‐shops of major chain supermarkets in Hong Kong. Data from the e‐shops of four major chain supermarkets were collected using a bespoke web scraper program, which extracted the product page link, product name, brand, package size, ingredient list in text and product image links of every edible product available. A total of 22 365 products which were required by law to carry a food label on their packaging were included in the analysis. Prevalence of missing nutrition label or ingredients list on the respective product page was determined. Pearson's χ2 tests were used to test for differences in the prevalence between retailers. A two‐tailed p < 0.05 was considered statistically significant. Two e‐shops provided nutrition labels and ingredients list for none or only a negligible number of products. Overall, the prevalence of missing information was higher for nutrition labels than for ingredients lists (80.5% vs. 70.2%; p < 0.001). Provision of information varied greatly between stores and food categories. We found a high prevalence of missing nutrition labels and ingredients lists on e‐shops of major supermarkets in Hong Kong. Regulations on nutrition labelling should be extended to cover pre‐packaged food and beverages sold online.

Web Scraper Utilizes Google Street View Images to Power a University Tour

Due to the outbreak of the Covid-19 pandemic, college tours are no longer available, so many students have lost the opportunity to see their dream school’s campus. To solve this problem, we developed a product called “Virtourgo,” a university virtual tour website that uses Google Street View images gathered from a web scraper allowing students to see what college campuses are like even when tours are unavailable during the pandemic. The project consists of 3/4 parts: the web scraper script, the GitHub server, the Google Domains DNS Server, and the HTML files. Some challenges we met include scraping repeated pictures and letting the HTML dropdown menu jump to the correct location. We solved these by implementing Python and Javascript functions that specifically target such challenges. Finally, after experimenting with all the functions of the web scraper and website, we confirmed that it works as expected and can scrape and deliver tours of any university campus or public buildings we want.

The online marketplace for business education: An exploratory study

The Internet has had a vast and pervasive impact on the education industry, one of which is the creation of an online educational marketplace. This study investigates the online marketplace for business courses by viewing it as part of the platform economy. Data pertaining to 9064 business courses was collected from the Udemy website using a web scraper tool. The analysis of the data reveals that the demand for MOOCs in business is great, with an average of 3169 students enrolled in each course, with some courses surpassing 100,000 students. The demand varies between categories, with business & data analytics, and entrepreneurship being among the most popular categories. The results also show that course enrollment in courses is highly skewed, giving support for the notion that power asymmetries that seem to arise in such decentralized markets. Finally, the analysis reveals that not all participants in the online marketplace for business education benefit equally. Specifically, the study found that courses tended to have a higher enrollment rate when the course provider was male. Finally, the analysis showed that despite increasing academic interest in online education, academics have been slow to enter the digital marketplace as service providers.

Consumer perceptions of hybrid electric vehicle adoption and the green automotive market: the Malaysian evidence

Compared to many developing countries, Malaysia has been far less successful in getting its citizen to embrace hybrid electric vehicles (HEVs). The study investigates the reasons behind this problem and offers a framework that could facilitate policymakers and stakeholders toward the electrification transition. Using the textual-based content analysis method, text-based articles and comments were extracted from a local reputable automotive news portal using a web scraper tool and analyzed using the open-coding technique. Surprisingly, environmental and greenhouse pollution issues did not emerge from our analysis—indicating that altruistic values and ecological consideration are not among the consumers' priorities when evaluating HEVs. Value proposition and risk aversion form the main themes, while national carmakers, competition, ambiguous policies, perceived unfairness in incentives distribution, and physical presentation constitute the minor themes. Overall, the consumers remain highly cautious and skeptical toward HEVs ownership proposition. This paper contributes to the existing literature by presenting a strategic framework for consumers' adoption of HEVs across Southeast Asian Nations (ASEAN). The main message is that HEV adoption requires a firm political resolution from ASEAN governments to promote a fairer playing field within the region by complementing (rather than competing with) each other. By removing non-tariff barriers and providing incentives, the gradual growth of a sizeable and sustainable HEV customer base will potentially dispel existing consumers' negative perceptions toward these cars.

An Approach to Analysis on Covid-19 Data Through Web Scraper and Voice Assistant – A Survey

Corona virus pandemic has been recognized as a global threat across the world and many methods were adopted for the prevention of this disease. This pandemic has caused global and economic disruption which resulted in numerous Covid-19 cases across the world. To know the number of cases and keep a track of this pandemic situation we need to collect the live data sets from the worldwide corona virus records. This can be achieved by the technique of Web Scraping which enables the extraction of live data sets from a specific platform. It facilitates the user to access the World Wide Web wherein specific data is gathered, copied from the web and then it is stored in a central local database then provides ways to retrieve and analyse the data. This estimate is to design a platform where you can obtain the live data sets and have a compact knowledge about the present scenario. This is an elementary approach to scrap the live data sets through a user interface from the Worldometer Covid-19 data set with the aid of a Google voice assistant. To implement this scheme, we use the Python programming language. To effectuate this task, we acquire the process of making API calls to the Worldometer Covid-19 website and simultaneously we will make use of the regular expressions to extract the data from the web page. However, this action includes a series of tactics that has to be recognized analysed and accomplished sequentially. Initially the input is given by the user in the form of speech. Then the required contents are searched and matched with the user’s input. If the contents match then with the help of a Google voice assistant result is obtained which is the output in turn.

Knowledge Marketplaces: An Analysis of the Influence of Business Models on Instructors’ Motivations and Strategies

Unlike MOOC platforms such as Coursera or edX, which typically partner with institutions of higher education, online knowledge marketplaces allow anyone to broadcast courses and charge for them. In this article, we investigate, through a mixed-method approach, the motivations and strategies of the instructors of Udemy and Skillshare. Semi-structured interviews and a quantitative analysis of the characteristics of Skillshare’s courses, obtained using a Web scraper, suggest that while a significant proportion of the marketplace’s instructors are outreach driven, the majority are income driven. They develop strategies to maximize their revenues, notably by adapting the characteristics of their courses, such as the number of videos, to the business model of the platform. Courses are shorter on Skillshare than on Udemy, where instructors’ incomes are proportional to the number of registrations. We hypothesize that the latter platform’s business model incentivizes instructors to create longer courses in order to attract wider audiences.