Web Crawler Research Articles

Prediction of accessibility testing using a generalized linear model for e-government

Using the United Nations’ Online Services Indicator (OSI) as a benchmark, the study analyzes Jordan’s e-government performance trends from 2008 to 2022, revealing temporal variations and areas of discontent. The research incorporates diverse testing strategies, considering technological, organizational, and environmental factors, and aligns with global frameworks emphasizing usability, accessibility, and security. The proposed model unfolds in three stages: data collection, performing data operations, and target selection using the Generalized Linear Model (GLM). Leveraging web crawling techniques, the data collection process extracts structured information from the Jordanian e-government portal. Results demonstrate the model’s efficacy in assessing accessibility and predicting web crawler behavior, providing valuable insights for policymakers and officials. This model serves as a practical tool for the enhancement of e-government services, addressing citizen concerns and improving overall service quality in Jordan and beyond.

Evolution of green travel behaviour on dynamic social networks

Encouraging residents’ green travel behaviour can reduce carbon emissions. However, existing research focuses more on the individual level than on green travel in the context of group interactions. This study aims to connect individual- and group-level insights by integrating the theory of planned behaviour (TPB) and the norm activation model (NAM). Based on empirical data from a questionnaire survey and web crawler, a simulation of the evolution of green travel behaviour interactions in social networks was conducted. The results reveal that the theory of planned behaviour and the norm activation model (TPB −NAM) can explain and predict green travel behaviour. With individuals’ interactions, social networks present scale-free characteristics, and the state of green travel behaviour tends to be stable. Lowering the opinion-bounded confidence of agents in social networks could promote green travel behaviour. This study extends the literature on the theory of green travel behaviour in terms of model integration and interactive decision-making between individuals and groups. The results support the promotion of residents’ green travel behaviour.

SAW-YOLO: A Multi-Scale YOLO for Small Target Citrus Pests Detection

Citrus pests pose a major threat to both citrus yield and fruit quality. The early prevention of pests is essential for sustainable citrus cultivation, cost savings, and the reduction of environmental pollution. Despite the increasing application of deep learning techniques in agriculture, the performance of existing models for small target detection of citrus pests is limited, mainly in terms of information bottlenecks that occur during the transfer of information. This hinders its effectiveness in fully automating the detection of citrus pests. In this study, a new approach was introduced to overcome these limitations. Firstly, a comprehensive large-scale dataset named IP-CitrusPests13 was developed, encompassing 13 distinct citrus pest categories. This dataset was amalgamated from IP102 and web crawlers, serving as a fundamental resource for precision-oriented pest detection tasks in citrus farming. Web crawlers can supplement information on various forms of pests and changes in pest size. Using this comprehensive dataset, we employed the SPD Module in the backbone network to preserve fine-grained information and prevent the model from losing important information as the depth increased. In addition, we introduced the AFFD Head detection module into the YOLOv8 architecture, which has two important functions that effectively integrate shallow and deep information to improve the learning ability of the model. Optimizing the bounding box loss function to WIoU v3 (Wise-IoU v3), which focuses on medium-quality anchor frames, sped up the convergence of the network. Experimental evaluation on a test set showed that the proposed SAW-YOLO (SPD Module, AFFD, WIoU v3) model achieved an average accuracy of 90.3%, which is 3.3% higher than the benchmark YOLOv8n model. Without any significant enlargement in the model size, state-of-the-art (SOTA) performance can be achieved in small target detection. To validate the robustness of the model against pests of various sizes, the SAW-YOLO model showed improved detection performance on all three scales of pests, significantly reducing the rate of missed detections. Our experimental results show that the SAW-YOLO model performs well in the detection of multiple pest classes in citrus orchards, helping to advance smart planting practices in the citrus industry.

OSTIS: A novel Organization-Specific Threat Intelligence System

With the increasing complexity and frequency of cyber attacks, organizations recognize the need for a proactive and targeted approach to safeguard their digital assets and operations. Every industry faces a distinct array of threats shaped by factors such as its industrial objective, geographic footprint, workforce size, revenue, partnerships, and the extent of its digital assets. This results in a wide heterogeneity in threat landscapes, which necessitates tailored threat intelligence sources. While some security practitioners may gravitate towards extensive sources, relying solely on volume-based solutions often leads to “alert fatigue”. For this reason, organization-specific threat intelligence has acquired a growing importance in cybersecurity defense.This work presents a complete and novel framework called OSTIS (Organization-Specific Threat Intelligence System) for generating and managing organization-specific Cyber Threat Intelligence (CTI) data. Our approach identifies reliable security blogs from which we gather CTI data through a custom and focused Web Crawler. Relevant content from such sources is, then, identified and extracted using automated deep-learning models. Moreover, our AI-driven solution maps CTI data to specific domain scenarios, such as education, finance, government, healthcare, industrial control systems, and IoT. To validate and gain insights from the trained models, we also include an explainable AI (XAI, for short) task carried out by leveraging the SHapley Additive exPlanations (SHAP) tool. This allows us to interpret the prediction process and discern influential content from data. The last step of our framework consists of the generation of an Organization Specific Threat Intelligence Knowledge Graph (OSTIKG), empowering organizations to identify and visualize attack patterns and incidents, promptly. To create this graph, we develop and adapt several techniques to extract diverse entities, including malware groups, campaigns, attack types, malware types, software tools, and so forth, and to identify relationships among them. Finally, through an extensive experimental campaign, we certify the validity and performance of all the components of our framework, which shows a 0.84 F1-score in the identification of relevant content, a 0.93 F1-score for the domain classification, and a 0.95 and 0.89 F1-score in the identification of entities and relations to build our OSTIKG graph.

Automated Ruleset Generation for “HTTPS Everywhere”

This paper details the implementation of a Web crawler aimed at automating ruleset construction for “HTTPS Everywhere,” with a goal to convert HTTP URLs to secure HTTPS equivalents for enhanced communication security. Developed within a seven-month timeframe, the crawler faced challenges in verifying HTTPS support, varying based on SSL certificate existence and validity. Successful ruleset creation and testing in Firefox and Chrome, adhering to stylistic standards, demonstrated the potential for effective development. The paper explores improving productivity through alternative libraries like Scrapy and Scrapy Cloud. While certain goals, such as in-depth cryptocurrency analysis and web crawler background reading, were unmet due to time constraints, valuable insights were gained. The conclusion underscores the difficulties, successes, and promises of automating ruleset generation through web crawlers for “HTTPS Everywhere,” offering valuable recommendations for advancing web security.

From Urban heritage resources: social media analytics of Johor Bahru tourists’ experience and motivation for exploring heritage buildings

ABSTRACT Assessing tourists’ social media perceptions is crucial in today's digital age. Due to its industrial reputation, Johor Bahru, Malaysia, has been overlooked for heritage tourism. This data mining study examines tourists’ experiences and motivations for visiting Johor Bahru's heritage buildings of 2,369 Google Review and Tripadvisor comments using web crawlers scripted in Python. The content analysis identified six themes associated with tourist motivation: activity, accessibility, diversity, attachment, inclusivity, and appearance. Even though their aesthetics are subdued, Johor Bahru's public historic sites thrive on accessibility and active activities, unlike magnificent royal edifices. Despite their architectural value, royal structures attracted fewer tourists due to the absence of diversity, inclusivity, and activities and restrictive measures. Maintaining the city's legacy and architectural identity requires highlighting these buildings. This study accentuates social media-driven popular views. The findings may encourage local authorities and stakeholders to strengthen Johor Bahru's identity and promote heritage tourism.

The status of the AI medical industry in China: A database and statistical analysis

BackgroundArtificial intelligence (AI) technology has developed rapidly in recent years, leading to exponential growth in the AI medical industry. However, a comprehensive investigation of approved medical devices in China is needed. MethodsWe utilized a web crawler to collect data on all medical devices from the China National Medical Products Administration website since 2018. Through natural language processing techniques and manual analysis, we identified all medical devices developed by artificial intelligence medical devices (AIMD) companies and conducted a statistical analysis. ResultsSince 2018, the number of AI-related medical devices approved in China has significantly increased. Most devices (79 %) were classified as Class II with moderate risk, whereas 21 % were classified as Class III with high risk. Most devices (74.2 %) were categorized as medical device software, and the most common application was medical image processing (63.2 %). In terms of target body areas, devices related to the heart accounted for the highest proportion (12.8 %), followed by those related to the lungs (11.3 %) and brain (6.7 %). ConclusionThis study establishes a comprehensive database of medical devices developed by AIMD companies in China, enabling the public to gain a coherent understanding of their current development status.

Optimizing the Implementation of the BFS and DFS algorithms using the web crawler method on the kumparan site

Efficient access to timely information is critical in today's digital era. Web crawlers, automated programs that navigate the Internet, play an important role in collecting data from websites such as Kumparan, a leading news site in Indonesia. This research shows the effectiveness of the Breadth-First Search (BFS) and Depth-First Search (DFS) algorithms in indexing Kumparan content. The results of the research show that BFS consistently indexes more files comprehensively but with longer execution times compared to DFS, which provides faster initial results but with fewer files. For example, at depth 4 BFS indexed 949 files in 886.94 seconds, while DFS indexed 470 files in 233.02 seconds. These findings highlight the balance between precision and speed when selecting a crawling algorithm tailored to the needs of a particular website. This research provides insights into optimizing web crawler technology for complex websites such as Coil and suggests avenues for further research to improve permission efficiency and adaptability across a variety of crawling scenarios.

Impact of tourism destination image on perceived usefulness of online reviews: Psychological distance effect and two methodologies

AbstractThis study aimed to empirically verify the impact of tourism destination image on perceived usefulness from the perspective of construal‐level theory. It also attempted to compare the results of the survey and data mining techniques regarding methodology. An online survey was initially conducted, and statistical analysis was performed using the R program. Concurrently, 25,952 reviews from TripAdvisor were gathered using a web crawler, Octoparse, and analyzed using the Tidyverse approach. The results indicated that, in the probabilistically distant group, affective destination image had a significant effect on perceived usefulness, whereas cognitive image did not affect perceived usefulness. Contrastingly, in the probabilistically close group, cognitive images had a stronger effect on perceived usefulness than affective images. Both the influential relationship between affective and cognitive images and the comparison of the two research methods were partially supported. These findings have theoretical and practical implications.

What Chinese Women Seek in Mental Health Apps: Insights from Analyzing Xiaohongshu User Posts during the COVID-19 Pandemic.

Gender disparity poses a prominent obstacle to achieving effective mental health outcomes in digital healthcare. Despite women being more inclined to use mental health apps and seeking designs tailored to their specific needs, there is limited research on the factors influencing female users' engagement with these apps. The COVID-19 pandemic has further exacerbated its disproportionate impact on women's mental health. This study investigates female users' posts (n = 5538) about mental health apps during the pandemic, using data collected via a Python web crawler from Xiaohongshu, a popular female-centric social media platform in China. A mixed-methods approach used qualitative thematic analysis and quantitative descriptive statistics. Among these posts, therapeutic functionality emerged as the highest priority, followed by credibility and user experience, with specific design elements highlighted as particularly significant. These findings provide valuable insights for mental health researchers and developers, including you, aiming to create gender-tailored mobile solutions to address the mental health challenges faced by women, especially during future pandemics.

Exploring the Chinese public's affective attitudes towards digital transformation in agriculture: A social media-based analysis.

This study utilizes natural language processing techniques and panel vector autoregression methodology, to delve into the perceived attitudes of social media users towards the digital transformation of agriculture, and to assess its impact on total agricultural output and agricultural science and technology inputs. Data related to agricultural digital transformation were collected from Sina Weibo using web crawlers. The SnowNLP model was employed to infer users' attitudes, encompassing both positive and negative aspects. Furthermore, the study delves into the specific themes capturing users' positive attitudes and explores regional variations in focus. The findings reveal a sustained increase in users' interest in agricultural digital transformation since 2013. Positive attitudes primarily center around green development, agricultural intelligence, and global cooperation and innovation. Moreover, the study establishes a significant positive impact of users' positive attitudes on both total agricultural output value and agricultural science and technology investment, highlighting the constructive influence of user support on the agricultural industry's development.

Spatiotemporal characteristics of residents' intercity travel in China under the impact of COVID-19 pandemic

Investigating the spatiotemporal characteristics of residents' intercity travel affected by the coronavirus disease (COVID-19) pandemic is of great significance to understanding and extending the impact of major public health affairs on intercity travelers, urban management policies, and intercity cooperative interactions. Taking 366 cities in China as an example, this study proposed a novel research framework for analyzing the spatiotemporal characteristics of intercity population flows. Using AutoNavi population migration big data sets from web crawling, the spatiotemporal structure patterns of intercity travel in China from January 1st to June 30th, 2020 were extracted by using the singular value decomposition (SVD) method. Then the spatial organization patterns of intercity travel networks in China and its five major urban agglomerations were identified by the weighted stochastic block model (WSBM). The results showed that the total volume of intercity travel in China and its five major urban agglomerations experienced a process of rapid decline followed by a slow recovery, presenting a “V-shaped” distribution since the outbreak of the COVID-19 pandemic. Three distinctive intercity travel patterns, namely the stable, holiday, and weekday-weekend intercity trip patterns, were extracted from the spatiotemporal OD matrix of intercity travel. Strong intercity travels were mainly distributed in eastern and central China, forming an intercity travel pattern in which core cities pushed and pulled each other within the national urban agglomerations. However, intercity migration flows were dominated by weak-intensity flows. Intercity trips during the holidays had a phenomenon of “escaping from the metropolis”. Meanwhile, it had significant spatial proximity and directionality. The daily intercity trips were characterized by a remarkable periodicity with seven days as a repeat. A more interesting finding is that when weekends overlap with holidays, intercity travel on the 1–2 days before the vacations has presented an effect of “suppression travel”. Furthermore, at the national scale, the stable intercity travel network has formed a polycentric core-periphery structure of “Three Cores-Six Semi-peripheries-Five Peripheries”. At the urban agglomeration scale, they formed respectively distinguished monocentric, bicentric, and polycentric core-periphery structures.

Research on influencing factors of pharmaceutical e-commerce sales based on web crawler and support vector machine

Research on influencing factors of pharmaceutical e-commerce sales based on web crawler and support vector machine

Application of High-Dimensional Data Visualization and Visual Communication Technology in Virtual Reality Environment

How to efficiently find what you want from massive data has become a hot topic. It is precisely because network information is developing so fast that so many data-oriented software systems and pages are used to manage so much network information. Aiming at problems such as insufficient information mining of visual communication websites in ASPNET and poor visual communication effects, a design scheme for integrating high-dimensional data into visual communication websites is proposed. Its working architecture includes high-dimensional data integration, multi-type visual integration, and visual support services. The main functions of the website include comprehensive management, charts, data sources, projects, systems, etc. We have conducted specific design and Implementation of high-dimensional information integration on the website, completed the design and Implementation of integration of high-dimensional data, integration of multiple visualization types, and integrated access to visualization-supported functions. Use a deep web crawler to search the visual site. It uses a search-based approach to filter URLs and crawl valuable web pages. Practice has proved that the intelligent distribution network management page implemented by the solution proposed in this article has good performance. The system shows good performance in visualization implementation and the ability to discover information. The system achieved sound and visual communication effects.

Immersive E-learning mode application in Chinese language teaching system based on big data recommendation algorithm

With the rapid development of information technology, E-Learning has become an innovative teaching method. However, in the field of Chinese teaching, how to provide effective learning resources and recommendation algorithms in E-Learning mode is still a challenge. This study aims to improve the effectiveness of Chinese teaching system and students’ learning outcomes through an immersive E-Learning model based on big data interest recommendation algorithm. This paper adopts an immersive E-Learning model based on big data interest recommendation algorithm, and constructs a Chinese teaching system. The web crawler is used to fully collect the experimental data and collate it in a targeted manner, and the required data is screened out by using a more efficient separation method. Adding big data recommendation algorithm to the system of this paper can not only record and analyze historical behaviors of users, but also recommend data information according to users’ interests, so that users can clarify their real information needs. By testing the system’s professional ability and recording the experimental data, this paper finds that the overall performance of this Chinese language teaching system is very good, and can achieve the original expected design purpose. In addition, the system largely solves the problem that the traditional system based on data recommendation algorithm is difficult to carry out effective recommendation smoothly when the total amount of data is too large.

Citizen science resource mobilization: Social identities and textual narcissism

Narcissism, as a form of self-awareness of self-importance or self-influence, can be characterized by a constant attention to success and the need for authority, competitiveness, and grandiose, which may be introduced to promote the attractiveness of a project. This study analyzes the effect of textual narcissism on the willingness to back citizen science projects by considering the social roles of the citizen project owners: expert, student, and amateur researchers. This paper uses social role theory to anchor the seven forms of narcissism and proposes 9 dimensions of narcissism, which are then classified into 3 categories: social role advantage, psychological superiority, and identity aspiration. Using a web crawler, 850 citizen science projects are employed as a corpus. Text mining is used to quantify the narratives, and econometric models are applied to estimate the effect of textual narcissism. This study reports that narcissism presents an inverted U-shaped effect. On social role, formal researchers (experts and students) receive lesser degrees of public tolerance to textual narcissism, over amateur researchers. By extending narcissistic influence and social role theory to citizen science, this paper can guide stakeholders and regulators such as scientific experiment platforms in generating suitable text descriptions for better research resource mobilization.

Adaptive CAPTCHA: A CRNN-Based Text CAPTCHA Solver with Adaptive Fusion Filter Networks

Text-based CAPTCHAs remain the most widely adopted security scheme, which is the first barrier to securing websites. Deep learning methods, especially Convolutional Neural Networks (CNNs), are the mainstream approach for text CAPTCHA recognition and are widely used in CAPTCHA vulnerability assessment and data collection. However, verification code recognizers are mostly deployed on the CPU platform as part of a web crawler and security assessment; they are required to have both low complexity and high recognition accuracy. Due to the specifically designed anti-attack mechanisms like noise, interference, geometric deformation, twisting, rotation, and character adhesion in text CAPTCHAs, some characters are difficult to efficiently identify with high accuracy in these complex CAPTCHA images. This paper proposed a recognition model named Adaptive CAPTCHA with a CNN combined with an RNN (CRNN) module and trainable Adaptive Fusion Filtering Networks (AFFN), which effectively handle the interference and learn the correlation between characters in CAPTCHAs to enhance recognition accuracy. Experimental results on two datasets of different complexities show that, compared with the baseline model Deep CAPTCHA, the number of parameters of our proposed model is reduced by about 70%, and the recognition accuracy is improved by more than 10 percentage points in the two datasets. In addition, the proposed model has a faster training convergence speed. Compared with several of the latest models, the model proposed by the study also has better comprehensive performance.

Exploring the Path of Personal Information Infringement Governance by Web Crawlers

In the era of big data, the explosive growth of data information, network crawler as a new type of data search engine came into being, and gradually become indispensable technology. However, with the continuous penetration of crawler technology, related infringement problems have gradually surfaced. This topic starts from the definition and technical application of network crawlers, further analyzes the specific forms of network crawlers’ personal information infringement, and discusses and summarizes the shortcomings of the legal governance measures involved, and finally draws on the experience of overseas and combines the views of relevant scholars, proposes to improve the relevant legal provisions and clarify the division of responsibility; introduces the rule of balanced trial; establishes the criteria for the reasonable use of data by crawlers; creates a “cyclical protection system”; establishes the “data protection system”; and establishes the “data protection system”. The proposal to establish a cyclical protection system, establish a “data protection officer”, set up an independent regulatory body, form an internal and external linkage mechanism, and then build a scientific governance system for scientific governance.

Evaluating Machine Learning Techniques for Web Robot Detection

A web robot, also known as a web crawler, is an automated script that systematically browses the World Wide Web in a methodical manner. Web robots are commonly used by search engines to index web pages, gather information, and update search engine databases. They follow hyperlinks from one web page to another, collecting data and information for various purposes such as indexing, data mining, and website monitoring. Web robot detection is the process of identifying and distinguishing between human users and web robots on websites, which are a major source of web traffic. This process is vital to prevent malicious web robots from having a negative effect on web servers’ traffic and their users’ privacy. Companies such as Imperva, Inc. use machine learning models to identify malicious bots and prevent them from having unauthorized access to an organization’s server. To understand the benefit that machine learning models bring to web robot detection, we used pre-published server log data to construct machine learning models on Orange (a Data Mining Software) and Python that can distinguish between malicious and benign web robots. We evaluated the performance of three well-known machine learning algorithms: kNN, neural network, and decision tree. Based on our experimental results, the neural network gains the highest precision and the lowest false-positive and false-negative percentage of web robots. However, the neural network requires more time to generate the desired output.

Research on Knowledge Graph Construction Methods for News Domain

With the rapid development of the Internet and artificial intelligence technology, massive text data is growing exponentially, and the news industry is generating massive text data every moment. Most of the existing knowledge graphs focus on tourism, healthcare, finance and other fields, while there are relatively few in the news field, which is not well constructed. Aiming at the above problems, this paper firstly devotes to the construction of ontology library in news domain, and precisely defines 8 types of entities and 9 types of relations, which lays a solid foundation for the construction of news knowledge graph. Subsequently, with the help of web crawler technology, we extensively collected news text and image data, and carried out rigorous knowledge cleaning, extraction and fusion processing on these data to ensure the accuracy and completeness of the data. Finally, with the help of Neo4j graph database, the effective storage of news knowledge is realised, and the news domain knowledge graph is successfully constructed. It provides new ideas and means for information mining and utilisation in the news industry, and also provides rich, high-quality data support for downstream applications, which is expected to promote the intelligent development of the news field.