Web Crawler Research Articles

Text analysis and optimization strategy of an app’s privacy policy from the perspective of dual perception: taking Chinese shopping apps as an example

Purpose This paper aims to conduct an in-depth analysis of the shortcomings of apps’ privacy policies and to propose improvement and optimization strategies, which are of great significance for establishing a transparent and responsible privacy protection framework that ensures compliant collection and use of users’ information and effective protection of their privacy. Design/methodology/approach This paper obtained privacy policy texts for 100 shopping apps through Web crawlers and manual downloads. Based on the perspective of perceived usefulness, thematic analysis is conducted through the latent Dirichlet allocation topic model and comparison with existing policies. Based on the perspective of perceived ease of use, readability analysis is conducted through content analysis and formula calculation. Findings The apps privacy policies can be divided into seven themes. The authors benchmark these seven topics with the Personal Information Protection Law of the People’s Republic of China, the E-Commerce Law of the People’s Republic of China and the General Data Protection Regulation. It is found that there are omissions in the information collection and use and juvenile protection of the existing apps. Through the indicators’ readability analysis and calculation, it is found that the existing apps privacy policies have good performance in the readability indicators such as naming method, frame directory and so on. However, text personalization and text readability need to be improved and optimized. Originality/value At the theoretical level, this paper constructs a model from the dual perception perspectives of perceived usefulness and perceived ease of use and analyses the apps’ privacy policy texts at a fine-grained level. At the practical level, based on large-scale apps’ privacy policy text data, this paper conducts multi-dimensional research from theme analysis, authoritative law benchmarking analysis, content analysis and text readability calculation and analysis. At the same time, this paper identifies the current problems of apps’ privacy policies and puts forward countermeasure suggestions for their content improvement and optimization.

An unsupervised machine learning-based profile system of Chinese researchers

The construction of researcher profiles is crucial for modern research management and talent assessment. Given the decentralized nature of researcher information and evaluation challenges, we propose a profile system for Chinese researchers based on unsupervised machine learning and algorithms. This system builds comprehensive profiles based on researchers’ basic and behavior information dimensions. It employs Selenium and Web Crawler for real-time data retrieval from academic platforms, utilizes TF-IDF and BERT for expertise recognition, DTM for academic dynamics, and K-means clustering for profiling. The experimental results demonstrate that these methods are capable of more accurately mining the academic expertise of researchers and performing domain clustering scoring, thereby providing a scientific basis for the selection and academic evaluation of research talents. This interactive analysis system aims to provide an intuitive platform for profile construction and analysis.

A Product’s Kansei Appearance Design Method Based on Conditional-Controlled AI Image Generation

Accurately grasping users’ Kansei needs and rapidly transforming them into product design solutions are key factors in enhancing product competitiveness and sustainability. This paper proposes a product appearance design method based on Kansei engineering and AI image generation technology, integrating other approaches, with household indoor hydroponics as the research subject. First, the web crawler is used to obtain product image samples and user online reviews, and factor analysis (FA) is applied to quickly extract users’ Kansei needs. Second, product morphology is used to deconstruct and encode product appearances. Partial least squares regression (PLSR) is then employed to map and quantify the relationships between Kansei needs and design elements, yielding optimal design solutions and one-dimensional sketches. These sketches are subsequently used as controlled conditions in Stable Diffusion (SD), combined with a team-trained Lora model, to generate two-dimensional colored sketches in batches. Finally, evaluations verify that the generated design solutions are satisfactory and meet users’ Kansei needs. The results indicate that the proposed product appearance design method not only holds significant implications for the sustainable development of Kansei engineering in product design but also greatly enhances the efficiency of the design process, providing new insights into integrating new technologies and scientific research methods in the field of product design.

Exploring the digital gray zone of online medicinal markets emerging from search

AbstractThis explorative study investigates the emergence of gray zone markets from search engines amidst the global expansion of online markets. With the analytical approach of infrastructural inversion, we examine how the search infrastructure constructs access to a gray zone market including both authorized online pharmacies and unauthorized vendors. Using Sweden and Google Search as a case, we explore the online presence of three products (vitamin D, paracetamol, and Viagra), through search engine result page analysis, web crawling, and network analysis. Infrastructural inversion unveils the typically invisible mechanisms of search engines, considering user queries, algorithmic priorities, SEO practices, and pharmacy regulations. We find gray zones only emerge in searches for erectile disfunction medicinal products and information, where unauthorized vendors successfully competed for visibility in search engine rankings. A complex web of conditions can steer consumers toward gray zone markets, complicating the access to safe and regulated medicinal products. This can expose individuals to risks associated with unverified medicinal products, but also challenges the integrity of the online health information infrastructure.

Generation Z Satisfaction with Smart Homestays: ASCI and Web Crawler Insights from China

In the digital context, smart homestays have developed rapidly in the post-epidemic era and have become a new form of accommodation. Homestays are favored by many young people in China, such as those belonging to Generation Z. According to data concerning China’s national tourism and related reports, the demand for homestays has increased dramatically in recent years. Thus, we need to consider how to improve the smart homestay user experience. Based on the American Customer Satisfaction Index (ACSI) model, this study explores the factors that affect the user experience of smart homestays. An online survey of 370 respondents of Generation Z in China was conducted, followed by descriptive statistical analysis and hypothesis model validation using SPSS 26.0. The data show that among the five service variables (reservation, check-in, living, check-out, and information sharing), perceived value has a positive and positive impact on service variables in all aspects. Finally, machine learning is used for emotion text analysis, and the results show that users are biased towards smart homestays in the sentiment analysis of the comments. Although smart homestays have a certain amount of attention, there is still a lot of room for progress in technology and services. The purpose of this study is to improve and perfect the rules for making smart homestay service standards based on understanding the satisfaction of Generation Z when using smart homestays while also providing a theoretical basis and practical manuals for the industry to promote the development of the industry and improve user experience. Based on the research results of the above literature, it is imperative to carry out research on Generation Z, the main force of future consumption, especially in the field of artificial intelligence.

Application Status of Hadoop in Data Cloud Computing

This paper summarizes the Hadoop platform and its application in network public opinion analysis, and emphasizes the advantages of Hadoop in big data storage and processing, as well as the development of cloud computing technology and its importance in data collection, processing and analysis. This paper also discusses the core technologies based on Hadoop and Spark, including keyword retrieval, web crawler and text mining, which together constitute a comprehensive toolkit for analyzing and managing online public opinion.

An Ontology Based Web Crawler with a Near-Duplicate Detection System to Improve the Performance of a Web Crawler

Purpose: The aim of the study is to examine how an ontology-based web crawler with a near-duplicate detection system improves the performance of a web crawler. Methodology: The experiment was carried out using secondary data from a sample web site which was used since crawling is an endless process. Using these two approaches, the ontology web crawler would search for relevant searches according to the search query of the user while the near-duplicate detection system would eliminate redundant data. Findings: It was observed that ontology web crawler performed better and faster than a normal crawler. It takes less execution time to search the web than other web crawlers. This is due to the fact that web documents are being filtered by the ontology web crawler such that only relevant web documents are retrieved according to the search query of the user. The relevant documents are further filtered by a near-duplicate detection system by removing web pages that are duplicates of each other and also remove near-duplicate web documents. This further reduces the number of web pages retrieved by the web crawler. This model saves on storage space because of the reduced number of web pages retrieved as it takes care of irrelevant and redundant web pages searched. Unique Contribution to Theory, Practice and Policy: The study recommends that the model can be improved to be dynamic by adding new relations that is the crawler should search for web pages related to the search even if they don’t contain the keywords searched. Domains and concepts should be added when visiting new web pages. Standardization of weights needs to be done because as of now experts assign weights to terms according to the area of expertise and knowledge.

Computer-Assisted Online Learning of English Oral Pronunciation Based on DAE End-to-End Recurrent Neural Networks

With the development of globalization, learning a second language has received increasing attention from people. To improve English oral proficiency, a computer-aided online learning system for English oral pronunciation is studied. A denoising autoencoder is integrated into the system to create a simplified end-to-end recurrent neural network for pronunciation detection and diagnosis based on deep learning. The study first collected and preprocessed oral pronunciation data of English learners, including enhancing speech signals and reducing noise. Next, an RNN model with Long Short-Term Memory (LSTM) as the core was constructed to capture time series characteristics in pronunciation. And use DAE to extract features and reduce the influence of background noise to enhance the recognition of pronunciation features. At the same time, the study utilized web crawler technology to collect a large amount of oral pronunciation data from non-native English learners, and constructed an English oral corpus containing pronunciation errors. And in order to simulate real situations, white noise and pink noise were artificially added to the corpus in the study, and they were divided into training and testing sets in a ratio of 60% to 40%. The results showed that the classification accuracy of the system in the training and testing sets under white noise environment was 78.97% and 94.01%, respectively, and the classification accuracy in the pink noise environment was 76.19% and 94.03%, respectively. The system’s error detection accuracy in vowel and consonant pronunciation detection is 88.91% and 91.68%, respectively, and the error correction accuracy in vowel and consonant pronunciation detection is 90.67% and 91.96%, respectively. In summary, the research on computer-aided online learning of English oral pronunciation based on Denoising Auto Encoders end-to-end recurrent neural networks has effectively improved learning efficiency.

Topic identification and content analysis of internet medical policies under the background of Healthy China 2030.

This study aims to analyse the content of internet medical policies, grasp the inherent laws of the development of internet medical policies and provide references for improving the policy system. On the basis of web crawler technology, 436 internet medical policies issued by the central and local governments after the Healthy China Strategy was proposed were collected. The BERTopic model was used to extract topics, and a comprehensive analysis of China's internet medical policy texts was conducted through the analysis of topic content, topic hierarchy and topic clusters. A total of 27 topics were identified in China's internet medical policies, mainly focussing on five topic clusters: medical service regulation, elderly nursing and children's healthcare using traditional Chinese medicine, user safety guarantees, health education and communication and infectious disease treatment and recovery guidance. Different topic contents play a role in promoting the comprehensive and standardized development of internet medical services. However, there is still room for further improvement in policy integrity, consideration of multiple types of users, and implementation effectiveness. Continuous efforts are needed to optimize the effectiveness of policies.

Food for Thought in Malaysian Borneo: Data Mining and Sentiment Analysis of Tripadvisor Reviews for Online Destination Branding

Unique destination branding in tourism is crucial to gaining a competitive advantage. Local food is a significant component of a destination brand which Destination Management Organisations (DMOs) must integrate into their official marketing efforts. DMOs must monitor the popularity and perception of their food promotions against the competition. Online promotion and management are challenging due to the rise of user-generated content. Automatic data mining techniques were used to determine the extent to which tourist food choices on TripAdvisor matched foods promoted by DMOs and how distinct these choices were from a rival DMOs’ promotion. We compared online food promotion between the Sabah Tourism Board and the Sarawak Tourism Board. We developed a software system to automatically extract food branding business intelligence from TripAdvisor restaurant reviews. The application of web crawling and scraping technology was applied to extract data and use Sentiment Analysis for interpretation. Online foods promoted by DMOs for each region were found to contain only a few common dishes, but these were more popular than region-specific foods. Significantly different distributions of food choices were found for each region. Some potentially useful differences between foreign and domestic tourists and locals were also identified. Sentiment analysis revealed hidden information in reviews useful for potential food destination branding. Findings from TripAdvisor confirm practical suggestions for improving brand distinctiveness found in the literature. This study is the first to develop an actual system that DMOs could use to estimate the online popularity of their promoted foods and those of their competition.

스마트관광 척도개발: 컨벤션산업을 중심으로

Purpose: Based on the existing discussions on smart tourism, this study intends to develop a scale by using both qualitative and quantitative methods of the compositional dimensions and sub-elements of smart tourism in the convention industry. Through this, the concept and components of smart tourism in the convention are clearly defined and organized, and a theoretical frame that will be the basis for smart tourism-related research is presented. Methods: Based on web crawling, this study first performed a cluster analysis by matching data to the Nvivo program, and second, an FGI and expert opinion survey were conducted to derive an item pool through qualitative analysis. Third, using the derived item pool, exploratory factor analysis and confirmatory factor analysis were performed using the data collected through the survey to secure reliability and validity. Results: The final derived compositional dimensions were finally derived in five dimensions: technology integration, convenience, information, smart infrastructure, and reliability, and a total of 15 questions were derived, 3 questions each for the lower dimensions. Conclusion: The results of this study provide the following theoretical and practical implications. Theoretically, it laid the foundation for systematically understanding the convergence of smart tourism and the convention industry. In practice, it is possible to lay the foundation for providing personalized services by analyzing participant's preferences and behavioral data, thereby increasing participants' satisfaction and willingness to re-engage.

A study on the impact of investor sentiment on individual stocks based on text mining

This paper investigates the influence of investor sentiment on individual stock prices through text mining. By reviewing relevant theories and literature, we utilized web crawler technology to collect comment data from the Eastmoney Stock Bar, constructing a comprehensive index of investor sentiment and an index of investor sentiment for individual stocks. Employing the VAR model and Granger causality test, we analyzed the correlation and dynamic relationship between investor sentiment and stock price fluctuations. The study found that stocks of different types of financial institutions react differently to investor sentiment, and that investor sentiment significantly affects stock prices. This paper puts forward corresponding policy recommendations, which can help market investors make scientific decisions and promote the healthy development of the market.

Exploration of Teaching Reform of Data Collection and Web Crawler Course Based on PDCA Cycle

The course of Data Collection and Web Crawler is highly practical and students are generally interested in learning it. However, the traditional teacher-centered teaching model focuses on knowledge transfer, which cannot meet the social demand for big data talents. PDCA (Plan-Do-Check-Act) cycle is the ideological basis of total quality management, which accords with the inherent law of teaching quality management. With "students" as the core and based on PDCA cycle, this paper carries out reform design from four processes of Plan, Do, Check and Act. The four stages of PDCA cycle correspond to the design, implementation, evaluation and reflection of teaching. The “Plan” part includes the design of teaching objectives and the design of teaching content. The teaching objectives are divided into knowledge objectives, ability objectives and quality objectives, while the design of teaching content can be summarized as addition, subtraction, adjustment and integration. The “Do” part mainly adopts a variety of teaching methods, such as the combination of theory and experiment, the combination of online and offline, the combination of group and individual reports. The “Check” part adopts a variety of evaluation methods, including examination, operation and reporting, to comprehensively evaluate students' knowledge, abilities and qualities. The “Act” part is mainly about standardizing successful methods, documenting existing problems to avoid repeat offenses, and transferring unresolved problems to the next cycle. Under the new teaching mode, the teaching is student-centered, giving full play to students' subjective initiative, and improving their ability and quality while mastering knowledge.

An AI-Powered Product Identity Form Design Method Based on Shape Grammar and Kansei Engineering: Integrating Midjourney and Grey-AHP-QFD

Product Identity (PI) is a strategic instrument for enterprises to forge brand strength through New Product Development (NPD). Concurrently, facing increasingly fierce market competition, the NPD for consumer emotional requirements (CRs) has become a significant objective in enterprise research and development (R&D). The design of new product forms must ensure the continuity of PI and concurrently address the emotional needs of users. It demands a high level of experience from designers and significant investment in R&D. To solve this problem, a generative and quantitative design method powered by AI, based on Shape Grammar (SG) and Kansei Engineering (KE), is proposed. The specific method is as follows: Firstly, representative products for Morphological Analysis (MA) are selected, SG is applied to establish initial shapes and transformation rules, and prompts are input into Midjourney. This process generates conceptual sketches and iteratively refines them, resulting in a set of conceptual sketches that preserve the PI. Secondly, a web crawler mines online reviews to extract Kansei words. Factor Analysis (FA) clusters them into Kansei factors, and the Grey Analytic Hierarchy Process (G-AHP) calculates their grey weights. Thirdly, after analyzing the PI conceptual sketches for feature extraction, the features are integrated with CRs into the Quality Function Deployment (QFD) matrix. Experts evaluate the relationships using interval grey numbers, calculating the optimal ranking of PI Engineering Characteristics (PIECs). Finally, professional designers refine the selected sketches into 3D models and detailed designs. Using a Chinese brand as a case study, we have designed a female electric moped (E-moped) to fit the PI and users’ emotional needs. Through a questionnaire survey on the design scheme, we argue that the proposed innovative method is efficient, applicable, and effective in balancing the product form design of PI and user emotions.

Analysis of Guangzhou city image perception based on weibo text data (2019–2023)

With the popularization of smart mobile terminals and social media, a large amount of data containing textual information about the city has been generated on social media platforms, covering all areas of the city. This provides a new way for the study of comprehensive perception of city image. In the Internet era, users express their opinions about cities through social media platforms (e.g., Sina Weibo), and mining this information helps to understand the image of cities on mainstream social media and to target positive images to improve the competitiveness of the city's image. In this paper, 370,000 microblog messages related to “Guangzhou City” between 2019 and 2023 are collected using web crawler technology, and three typical text analysis methods are adopted: Term Frequency-Inverse Document Frequency (TF-IDF), Latent Dirichlet Allocation (LDA), and Sentiment Analysis (SnowNLP), to understand the characteristics of Guangzhou city image. gain an in-depth understanding of Guangzhou's urban image characteristics. The study shows that extensive data analysis methods based on text mining can perceive the dynamics and trends of the city in a timely manner, refine the characteristics of Guangzhou's urban image, and propose communication strategies for Guangzhou's image. This study aims to mine Guangzhou's urban image presented on Weibo, provide data support for relevant departments in China and Guangzhou to formulate communication strategies, and provide references for other cities to manage their urban image.

Attention-based fuzzy neural networks designed for early warning of financial crises of listed companies

Developing an early warning model for company financial crises holds critical significance in robust risk management and ensuring the enduring stability of the capital market. Although the existing research has achieved rich results, the disadvantages of insufficient text information mining and poor model performance still exist. To alleviate the problem of insufficient text information mining, we collect related financial and annual report data from 820 listed companies in mainland China from 2018 to 2023 by using sophisticated web crawlers and advanced text sentiment analysis technologies and using missing value interpolation, standardization, and data balancing to build multi-source datasets of companies. Ranking the feature importance of multi-source data promotes understanding the formation of financial crises for companies. In the meantime, a novel Attention-based Fuzzy Neural Network (AFNN) was proposed to parse multi-source data to forecast financial crises among listed companies. Experimental results indicate that AFNN exhibits significantly improved performance compared to other advanced methods.

Assessing Patient Satisfaction Following Facelifts with Social Media Reviews.

Facelifts are one of the most common facial aesthetic surgery procedures. Patient satisfaction determines success of most aesthetic surgery but has been historically difficult to assess. This study evaluated reviews by facelift patients on the aesthetic surgery social media website RealSelf.com to determine positive and negative factors underlying patient satisfaction following facelifts. Facelift reviews were gathered from RealSelf.com with an automated web crawler. Reviews were categorized as positive or negative and by the primary and secondary reasons for the positive or negative review. Patient "worth it" and star ratings, physician specialty, and cost of procedure were also collected. A total of 2153 facelift reviews were collected. Overall, 1986 (92.24%) were positive and 167 (7.76%) were negative. The most common overall reasons for a positive review were aesthetic results (n=1571, 79.10%) and bedside manner (n=1488, 74.92%). The most common overall reasons for a negative review were outcome (n=137, 82.04%) and bedside manner (n=82, 49.10%). Most facelifts were performed by plastic surgeons (n=1796, 83.42%). The greatest 5-star rating percentages were seen for oral and maxillofacial surgeons (n=29, 93.55%), otolaryngologists (n=96, 92.31%), and plastic surgeons (n=1642, 91.43%). Of patients who provided a "worth it" rating, 1216 (91.91%) stated that their facelift was "worth it." Overall patient sentiment toward facelifts was positive. The factors most commonly affecting a positive patient experience were bedside manner and aesthetic results. Negative patient reviews were primarily attributed to dissatisfaction with aesthetic outcomes. Social media serves as a valuable tool for evaluating patient satisfaction with aesthetic surgery. This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

Prediction of accessibility testing using a generalized linear model for e-government

Using the United Nations’ Online Services Indicator (OSI) as a benchmark, the study analyzes Jordan’s e-government performance trends from 2008 to 2022, revealing temporal variations and areas of discontent. The research incorporates diverse testing strategies, considering technological, organizational, and environmental factors, and aligns with global frameworks emphasizing usability, accessibility, and security. The proposed model unfolds in three stages: data collection, performing data operations, and target selection using the Generalized Linear Model (GLM). Leveraging web crawling techniques, the data collection process extracts structured information from the Jordanian e-government portal. Results demonstrate the model’s efficacy in assessing accessibility and predicting web crawler behavior, providing valuable insights for policymakers and officials. This model serves as a practical tool for the enhancement of e-government services, addressing citizen concerns and improving overall service quality in Jordan and beyond.

Evolution of green travel behaviour on dynamic social networks

Encouraging residents’ green travel behaviour can reduce carbon emissions. However, existing research focuses more on the individual level than on green travel in the context of group interactions. This study aims to connect individual- and group-level insights by integrating the theory of planned behaviour (TPB) and the norm activation model (NAM). Based on empirical data from a questionnaire survey and web crawler, a simulation of the evolution of green travel behaviour interactions in social networks was conducted. The results reveal that the theory of planned behaviour and the norm activation model (TPB −NAM) can explain and predict green travel behaviour. With individuals’ interactions, social networks present scale-free characteristics, and the state of green travel behaviour tends to be stable. Lowering the opinion-bounded confidence of agents in social networks could promote green travel behaviour. This study extends the literature on the theory of green travel behaviour in terms of model integration and interactive decision-making between individuals and groups. The results support the promotion of residents’ green travel behaviour.

SAW-YOLO: A Multi-Scale YOLO for Small Target Citrus Pests Detection

Citrus pests pose a major threat to both citrus yield and fruit quality. The early prevention of pests is essential for sustainable citrus cultivation, cost savings, and the reduction of environmental pollution. Despite the increasing application of deep learning techniques in agriculture, the performance of existing models for small target detection of citrus pests is limited, mainly in terms of information bottlenecks that occur during the transfer of information. This hinders its effectiveness in fully automating the detection of citrus pests. In this study, a new approach was introduced to overcome these limitations. Firstly, a comprehensive large-scale dataset named IP-CitrusPests13 was developed, encompassing 13 distinct citrus pest categories. This dataset was amalgamated from IP102 and web crawlers, serving as a fundamental resource for precision-oriented pest detection tasks in citrus farming. Web crawlers can supplement information on various forms of pests and changes in pest size. Using this comprehensive dataset, we employed the SPD Module in the backbone network to preserve fine-grained information and prevent the model from losing important information as the depth increased. In addition, we introduced the AFFD Head detection module into the YOLOv8 architecture, which has two important functions that effectively integrate shallow and deep information to improve the learning ability of the model. Optimizing the bounding box loss function to WIoU v3 (Wise-IoU v3), which focuses on medium-quality anchor frames, sped up the convergence of the network. Experimental evaluation on a test set showed that the proposed SAW-YOLO (SPD Module, AFFD, WIoU v3) model achieved an average accuracy of 90.3%, which is 3.3% higher than the benchmark YOLOv8n model. Without any significant enlargement in the model size, state-of-the-art (SOTA) performance can be achieved in small target detection. To validate the robustness of the model against pests of various sizes, the SAW-YOLO model showed improved detection performance on all three scales of pests, significantly reducing the rate of missed detections. Our experimental results show that the SAW-YOLO model performs well in the detection of multiple pest classes in citrus orchards, helping to advance smart planting practices in the citrus industry.