Data Crawler Research Articles

Retraction Note: Efficiently harvesting deep web interfaces based on adaptive learning using two-phase data crawler framework

Retraction Note: Efficiently harvesting deep web interfaces based on adaptive learning using two-phase data crawler framework

Crawler Data Visualization in the Tourism Industry Based on Python

Data visualization is being widely used in various fields, such as medicine, biology, economics, and other disciplines. In the tourism industry, whether the display of tourism resources, the planning of tourism routes, the feedback of tourists' experience, and so on, all of them become clearer and more intuitive through data visualization. In particular, Python-based data visualization has many advantages in the tourism industry. This paper mainly analyzes the current situation and advantages of Python-based data visualization in the tourism industry, and puts forward corresponding improvement measures and suggestions, aiming to help tourism practitioners and tourists use data visualization more conveniently and efficiently. It is found that data visualization plays an important role in displaying tourist attractions, planning tourist routes, and enhancing the experience of tourism services. However, the problems of data visualization in tourism, such as insufficient interactivity and single visualization style, require improvement measures. For example, methods such as optimizing algorithms and introducing interactive visualization tools can solve these problems. The purpose of this study is to promote a wider and more effective application of data visualization in the tourism industry so that tourism-related industries can be more efficient and intuitive when using data visualization tools.

Study on the Historical Evolutionary Path of Chinese Ceramic Technology Based on Data Mining

Abstract Ancient Chinese porcelain occupies an important position in Chinese cultural relics, and the development process of ceramics in various periods has played a great role in promoting the development of society. To explore the historical evolution path of Chinese ceramic technology, this paper proposes a data mining approach. The first step is to list the data mining tools, and then a distributed data crawler is deployed to crawl textual information related to Chinese ceramics in the Cultural Collection by defining keywords related to Chinese porcelain. Then, the incomplete data duplication, semantic ambiguity, and low-value and unclear pointing information that existed in the original text data were deleted. By setting thresholds and sorting, keywords with higher weights were filtered after scanning the cleaned texts. To compare data from different periods and subjects, the data was normalized. Correlation analysis was used to extract historical changes in Chinese ceramic technology in the time dimension, and then the components of ceramics in the historical evolution were analyzed by K-mean-based principal component analysis. Ceramic manufacturing technology mainly developed during the Five Dynasties and Two Songs, formed during the Yuan Dynasty, matured during the Ming Dynasty, flourished during the Qing Dynasty, and declined during the Republic of China. The first six principal components were extracted because their cumulative variance contribution exceeded 85%. In detail, the samples were divided into four categories according to each principal component, and the classification results reflect the differences in the content of pottery-making raw materials. Side by side, it reflects that pottery-making technology is changing with the advancement of history.

A Chinese power text classification algorithm based on deep active learning

The construction of knowledge graph is beneficial for grid production, electrical safety protection, fault diagnosis and traceability in an observable and controllable way. Highly-precision text classification algorithm is crucial to build a professional knowledge graph in power system. Unfortunately, there are a large number of poorly described and specialized texts in the power business system, and the amount of data containing valid labels in these texts is low. This will bring great challenges to improve the precision of text classification models. To offset the gap, we propose a classification algorithm for Chinese text in the power system based on deep active learning (CCTP-DAL). Our core idea is to apply a hierarchical confidence strategy to a deep active learning model, to balance the trade-offs between the amount of training data and the accuracy of text classification. Our CCTP-DAL (1) trains the Bert model using a small amount of labeled data to calculate the confidence level of each short text, (2) selects high-confidence text data with optimal model generalization capability based on the hierarchical confidence level, and (3) fuses deep learning models and active learning strategies to ensure high text classification accuracy with less labeled training data. We benchmark our model on a real crawler data on the web with extensive experiments. The experimental results demonstrate that our proposed model can achieve higher text classification accuracy with less labeled training data compared with other deep learning models.

A Specialized Data Crawler for Urban Wood Information

Abstract The Internet is composed of >50 billion Web pages and grows larger every day. As the number of links and specialty subject areas increases, it becomes ever more difficult to find pertinent information. For some subject areas, special purpose data crawlers continually search the Internet for specific information; examples include real estate, air travel, auto sales, and others. The use of such special purpose data crawlers (i.e., targeted crawlers and knowledge databases), also allows the collection and analysis of agricultural and forestry data. Such single-purpose crawlers can search for hundreds of keywords and use machine learning to determine whether or not what is found is relevant. In this paper, we examine the design and data return of such a specialty knowledge database and crawler system developed to find information related to urban wood utilization—products made from timber harvested in cities and municipalities. Our search engine uses intelligent software to locate and update pertinent references related to urban wood as well as to categorize information with respect to common application and interest areas. At the time of this publication, the urban wood knowledge database has cataloged >700 publications regarding various aspects of urban wood.

The Impact of College Students’ Perceived Service Quality in the Context of Regional Integration of Education

Exchange programs have been found to have a positive impact on regional economic growth, university cooperation, and student academic, and professional development. However, there has been limited analysis of the factors that influence the quality of these programs. In this study, a mixed-methods approach was used to examine regional exchange programs in China. Qualitative research was conducted using Octopus Big Data Crawler software, Timdream.org software, and 33 interviews to develop a questionnaire that covers student expectations, service quality, and satisfaction. Using SPSS 22 and Amos 21, the study analyzed the interactive relationships among expectations, perceived service quality, and satisfaction in the context of regional education integration based on a sample of 246 questionnaires. The results demonstrate that perceived service quality plays an intermediary role in the relationship between expectations and satisfaction, with students’ perceptions of learning quality and interpersonal quality at the receiving university influencing their satisfaction levels. This study provides important insights into the factors that impact the quality of regional exchange programs in China.

Target Category Agnostic Knowledge Distillation With Frequency-Domain Supervision

Existing knowledge distillation approaches require task-related data to train portable student networks for satisfactory performance. Nevertheless, in real-world applications, a majority of the data is unavailable over concerns about personal privacy, commercial confidentiality, etc. To mitigate the difficulty of acquiring the target dataset, the mainstream knowledge distillation methods generate training samples from the teacher network. However, the generated images still differ from the authentic ones, which limits the student network performance. To solve this issue, we propose a convenient and cost-negligible method to build datasets for training student networks. Specifically, we crawl the data from the web without knowing any category of the target dataset, named Irrelevant Category Crawler Data (ICCD). To prevent the performance collapse due to the data distribution gap between our ICCD and the target dataset, we propose a Pseudo-Classification strategy and Frequency-domain Supervision (PCFS) for target-category agnostic knowledge distillation with ICCD. The Pseudo-Classification strategy classifies ICCD into different pseudo-categories by the teacher network, and uniformly but randomly samples images from each pseudo-category to construct the pseudo-target dataset. Furthermore, we transform the feature maps from the spatial-domain to the frequency-domain, and utilize the high-frequency and low-frequency signals of the teacher network to impose strong constraints on the student network. Extensive experiments conducted on various test sets demonstrate the effectiveness of our proposed PCFS, which outperforms existing data-free methods and achieves comparable performance to those using the target training set. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/SCUT-BIP-Lab/PCFS-DFKD</uri> .

Mathematical model for portraits of members in large-scale department stores

Abstract The traditional retail industry be affected of the rapid development of modern e-commerce industry. To improve the shopping mall member portrait has become an effective way for operators to refine management and give full play to the value of members. We hope that the member system analysis based on data mining can provide reliable basis for establishing stable member relationship and planning promotional activities. In this paper, based on the real data provided by large department stores, adopting software such as MATLAB and SPSS, using probability statistics, data crawler and other related knowledge to description of a member of a large department store. Firstly, through the relationship models between members’ consumption characteristics model, it is concluded that the profit contribution of members to shopping malls is higher than that of non-members. Through the members’ purchasing power model, it is concluded that different strategies can be adopted for different levels of members. Through the members’ life cycle and state division model, we got the value of the member’s life cycle is about 799, and the state division threshold is 4.5594. Through the members’ preferences and commodity joint rate, we got that promotional activities play a leading role in the activation of members. Finally, according to the preferences of members, combined with business profits, we can choose the following three kinds of goods to promote: 30768d8b, d55deeb5, 6feea3f5.

THE OVERSEAS DISSEMINATION AND RECEPTION OF THE TRANSLATION OF THE QUOTATIONS OF MAO ZEDONG: BASED ON AMAZON READER REVIEWS

For a long time, as the main object of foreign translation and dissemination of Chinese books, The Quotations of Mao Zedong has had a remarkable impact on China and the world. The comments and evaluations of foreign readers are an important aspect to study and measure the effectiveness of the work’s foreign translation and distribution. In this paper, with the help of data crawler tool and text mining tool ROST CM6, a quantitative and qualitative study on the reader reviews of the translation of The Quotations of Mao Zedong on Amazon platform is conducted. The trends of star ratings, emotional attitudes and number of reviews show that most readers have positive attitudes toward it, and give it positive evaluations. Readers’ motivation for reading is based not only on circulation but also on personal interests such as historical, philosophical or study needs; at the same time, its political ideology is another important reason. Although there are flaws in the quality of book design and the language quality of the translation, the overseas dissemination of the translation of The Quotations of Mao Zedong is on the rise and the overall acceptance is good.

Data sharing for business model innovation in platform ecosystems: From private data to public good

Extant research posits that open data could unlock more than $3 trillion in additional value worldwide across various application domains. This paper investigates a data-sharing perspective in business models of platform ecosystems and discusses how platform owners can derive more value using data. We chose a sample of 12 platforms in which data are used as a key resource for service propositions. By contrasting these cases, we identify and analyse four archetypes: data crawler, data marketplace, data aggregator, and data disseminator. We define the key features of these archetypes and demonstrate how they realise value via the platform. These archetypes can guide managers in realising private and public goods via data sharing. Building on our findings, we derive recommendations for data-driven business model innovation for platform ecosystems.

More popular, more listings? Effects of popularity on Airbnb host expansion

PurposeSharing accommodation has lowered the threshold for digital entrepreneurship in the accommodation industry, prompting entrepreneurs to join this industry. However, digital micro-entrepreneurs have been ignored by previous studies. To bridge the gap, this study aims to explore the impact of reputational asset (host popularity) and host’s human capital (entrepreneurial entry speed and managerial seniority) on host expansion grounded on the resource-based theory.Design/methodology/approachThis study obtained crawler data by python from Airbnb.com, locking the time range to the past five years from 2013 to 2018 in Beijing of China. This study finally has 348 hosts’ balanced panel data to estimate the ordinary least squares regression model with fixed-year effect.FindingsResults demonstrate that host popularity has a significant positive effect on host expansion. Furthermore, entrepreneurial entry speed strengthens the positive effect of host popularity on host expansion, whereas managerial seniority weakens the positive effect. The three-way interaction analysis reveals that the positive impact of host popularity on host expansion is strongest when managerial seniority is smaller and entrepreneurial entry speed is faster.Research limitations/implicationsThe research has important implications to how the platforms interfere with the implementation of host expansion strategy and adds valuable insights to understand the transformation process of host expansion from nonprofessional to professional.Originality/valueThe research has expanded the literature related to the micro-entrepreneurship of the sharing economy and verified the application of resource-based theory under sharing economy.

A Mixed Methods Assessment of Self- Management Needs and Preferences of People with Type 2 Diabetes Mellitus in China.

Consistent and effective self-management is a major challenge for people with diabetes and long-term effects can be difficult to sustain despite the many interventions. Interventions often fail to take patients' perceptions of self-management into account from their perspective. Exploring the needs preferences and the influencing factors of self-management in patients can improve the effectiveness of self-management. This study used a mixed research approach, exploring patient needs and preferences through web crawl methods and using qualitative interviews to explore the logic of preferences. 1605 data were obtained from 4 online health communities through web crawlers. Data were coded using grounded theory to analyze self-management needs. The codes were also coded for frequency statistics and ranked to explain the self-management preferences. The second phase explained the logic of the ranked distribution of preferences through one-on-one interviews with 22 patients. This study summarized six self-management needs of people with diabetes through web crawler data, among which medication management and conflict between disease and daily life were the most important concerns of patients. The reasons why patients are particularly concerned about these two needs were explored through qualitative interviews, and it was found that patients' concerns about medication were mainly due to the following three aspects: the effect of drugs is direct and obvious, medication use makes life less complicated, and progressive side effects. The patients' concerns about the conflict between disease and daily life are often caused by the following three aspects: diabetes impacts social roles, disease complicates life, and reducing the impact of the disease on life. The self-management needs of Chinese diabetics have six themes. Medication management, conflict between disease and daily life, diet, diabetes knowledge, blood glucose monitoring and exerciseIn order of frequency of mention and possible preference: medication management, conflict between disease and daily life were the most frequent needs, while diet, knowledge about diabetes, blood glucose monitoring and exercise were their least frequent needs.Understanding the individual's perception and understanding of diabetes self-management and the expression of the patient's daily life situations can guide the medical team to optimize collaboration on personalized care plans.

Perspective of People With Type 2 Diabetes Toward Self-management: Qualitative Study Based on Web Crawler Data.

The diabetes disease burden in China is heavy, and medical standards such as diabetes guidelines are the core reference guidelines for diabetes management for health care providers and patients. However, patients' guideline compliance is too low, which correlates with the gap between guidelines and patients' self-management needs. Incorporating patient needs into the guideline development would reduce this gap. We sought to capture the needs of patients with diabetes for self-management in everyday situations and to clarify the contradictions and misalignments between medical standards, such as guidelines, and patient needs. This study collected crawler-based data from 4 online health communities. We selected 1605 text records collected from Chinese patients with diabetes between March 2020 and July 2020 for analysis. The text analysis applied grounded theory to separate issues that concerned patients into 3 themes, 7 subthemes, and 25 entries. Altogether, 69.03% (1108/1605) of texts were related to issues concerning disease treatment (theme B) and mainly inquired about medication use (B2 and B3; 686/1108, 61.91%), including medication choice, change in medication administration, side effects, and postmedication effects. In addition, 222 (N=1605, 13.8%) texts (theme A) concerned the explanation of disease etiology and knowledge of diabetes, and 275 (N=1605, 17.1%) texts (theme C) discussed lifestyle changes and various restrictions on life brought about by the disease. Our findings suggest an urgent need to improve diabetes health education and guideline development strategies and to develop health management strategies from a patient perspective to bridge the misalignment between patient needs and current medical standards.

Air Pollution Increased the Demand for Gym Sports under COVID-19: Evidence from Beijing, China.

Air pollution may change people’s gym sports behavior. To test this claim, first, we used big data crawler technology and ordinary least square (OLS) models to investigate the effect of air pollution on people’ gym visits in Beijing, China, especially under the COVID-19 pandemic of 2019–2020, and the results showed that a one-standard-deviation increase in PM2.5 concentration (fine particulate matter with diameters equal to or smaller than 2.5 μm) derived from the land use regression model (LUR) was positively associated with a 0.119 and a 0.171 standard-deviation increase in gym visits without or with consideration of the COVID-19 variable, respectively. Second, using spatial autocorrelation analysis and a series of spatial econometric models, we provided consistent evidence that the gym industry of Beijing had a strong spatial dependence, and PM2.5 and its spatial spillover effect had a positive impact on the demand for gym sports. Such a phenomenon offers us a new perspective that gym sports can be developed into an essential activity for the public due to this avoidance behavior regarding COVID-19 virus contact and pollution exposure.

The Influencing Factors of Customer Satisfaction of O2O Food Takeout Platforms – A Social Institutional Environment Perspective

Online-to-offline (O2O) takeout platforms have become an important way of food consumption in modern life. However, some problems, such as unlicensed operation and poor service, have been exposed. Therefore, it is very important to identify the factors that affect customer trust and satisfaction. However, there are very few studies on consumer satisfaction of takeout platforms in the existing literature. This paper builds a model according to the development of the O2O takeout industry and verifies it by questionnaires and web crawler data. Finally, SPSS data analysis software is used to analyze the variables. The results show that internal mechanisms and social environments affect the trust of users regarding O2O platforms. Also, with the improvement of users' trust, their perceived level of risk will decline, and their satisfaction will increase. Finally, suggestions of optimizing the platform mechanisms and improving service quality and attitude are advised.

Urbanization Influences CO2 Emissions in the Pearl River Delta: A Perspective of the “Space of Flows”

As the largest carbon emitter in the world, China is facing increasing challenge to reduce CO2 emissions. Given this issue, exploring the influencing factors is of great significance for scientific low-carbon emission policymaking. Although previous literature has explored the effects of urbanization on CO2 emissions, the impact of the space of flow on urban carbon emissions have been less explored. Due to the increasing connection between cities, its impact on urban carbon emissions cannot be ignored. Thus, this paper takes the space of flows into account as an aspect of urbanization to supplement the existing literature and empirically examines the multiple effects of urbanization on CO2 emissions in the Pearl River Delta (PRD) urban agglomeration. By using a STIRPAT model, statistical data, and web crawler data, we examined impacts of different types of urbanization on CO2 emissions. Our empirical results show that: (1) Within the PRD urban agglomeration, urban linkage intensity is strongly connected to urban socioeconomic growth, establishing a geographical structure with Guangzhou and Shenzhen as the double core. (2) Our results show that urbanization exerts two opposite effects on CO2 emissions: positively connects carbon emissions with population urbanization, integrated urban linkage flow, and energy intensity, whereas economic urbanization and social urbanization are shown to be negatively correlated. However, spatial urbanization has no significant positive effect on urban CO2 emissions. (3) It is worth noting that urban linkage flows are the second most important factor affecting urban carbon emissions after economic urbanization. Our study could formulate effective planning suggestions for future CO2 emission reduction paths and development modes in the PRD.

Development of CNN-Based Data Crawler to Support Learning Block Programming

Along with the importance of digital literacy, the need for SW(Software) education is steadily emerging. Programming education in public education targets a variety of learners from elementary school to high school. This study was conducted for the purpose of judging the proficiency of low school-age learners in programming education. To achieve the goal, a tool to collect data on the entire programming learning process was developed, and a machine learning model was implemented to judge the proficiency of learners based on the collected data. As a result of determining the proficiency of 20 learners, the model developed through this study showed an average accuracy of approximately 75%. Through the development of programming-related data collection tools and programming proficiency judging models for low school-age learners, this study is meaningful in that it presents basic data for providing learner-tailored feedback.

An Empirical Study on the Precise Employment Situation-Oriented Analysis of Digital-Driven Talents with Big Data Analysis

This paper conducts an in-depth research analysis on the precise employment of college graduates in the context of big data using a number-driven approach. The textual information of the study is obtained by using in-depth interviews, and the evaluation index system of college students’ employment quality is constructed by combining the step-by-step coding method with rooting theory. The research on the current situation of employment recommendation platform research and the application status of big data in the employment recommendation platform is explored by using a bibliometric approach. And the innovative use of web crawler technology is used to comprehensively understand the recommendation function and status quo of the same type of recommendation platform, which provides a reference for the research of this platform. Based on the preliminary analysis of platform requirements and overall design, the overall design and functional implementation of the big data employment recommendation platform are carried out by using big data crawler technology, big data architecture technology, text mining technology, database technology, etc. The construction of a recommendation module based on user history information, a recommendation based on real-time user online behavior data, and hybrid recommendation carried out on the recommendation module to grasp all-round the platform is built based on a stakeholder perspective. Based on the platform construction, the initial platform operation and maintenance management mechanism was established from the stakeholder’s perspective. The Pearson correlation coefficient is used to objectively evaluate the current situation of talent supply in universities and talent demand in enterprises from the perspective of image and data. In the research on the development status of the big data education industry, the Lorenz curve and Gini coefficient are used to match the status of new big data majors with their college construction volume in each province and provide data support for the reasonable adjustment of majors setting in each province according to the education level.

Two-Stage Robust Optimization Model for Uncertainty Investment Portfolio Problems

Investment portfolio can provide investors with a more robust financial management plan, but the uncertainty of its parameters is a key factor affecting performance. This paper conducts research on investment portfolios and constructs a two-stage mixed integer programming (TS-MIP) model, which comprehensively considers the five dimensions of profit, diversity, skewness, information entropy, and conditional value at risk. But the deterministic TS-MIP model cannot cope with the uncertainty. Therefore, this paper constructs a two-stage robust optimization (TS-RO) model by introducing robust optimization theory. In case experiments, data crawler technology is used to obtain actual data from real websites, and a variety of methods are used to verify the effectiveness of the proposed model in dealing with uncertainty. The comparison of models found that, compared with the traditional equal weight model, the investment benefits of the TS-MIP model and the TS-RO model proposed have been improved. Among them, the Sharpe ratio, Sortino ratio, and Treynor ratio have the largest increase of 19.30%, 8.25%, and 7.34%, respectively.

Mixed Land Use Evaluation and Its Impact on Housing Prices in Beijing Based on Multi-Source Big Data

The tense relationship between the supply and demand of land resources and the past spatial expansion of urban development in Beijing have brought many urban problems. Mixed land use is considered to be able to solve these urban problems as well as promote sustainable urban development. In this context, this study uses multi-source big data such as POI, OpenStreetMap and web crawler data to construct current land-use data of the area within the sixth ring road of Beijing, and then uses the entropy index and type number index to analyze the spatial distribution and aggregation characteristics of the mixed land-use level. Finally, a multi-scale geographically weighted regression is applied to explore the impact of the block and life circle scale mixed land use on housing prices. The results show that: (1) the accuracy of land use data obtained by using multi-source big data is high, and the consistency with the real land use situation is as high as 82.67%. (2) the mixed land use level in the study area is higher in the urban center and lower in the periphery of the city. However, it does not show the spatial distribution characteristics gradually decreasing with the increase of the distance from the urban center but shows that the area from the third to the fifth ring road is the highest. (3) the impact of block scale and life circle scale mixed land use on housing price is different. The type number index has a negative effect on the housing price in block scale mixed land use, while the entropy index has a positive effect on the housing price in life circle scale mixed land use. Based on the existing “bottom-up” individual-dominant development mode, the government of Beijing should issue relevant policies and documents to give “top-down” control and guidance in the future, so as to promote the maximization of the benefits of mixed land use. Furthermore, in the practice of mixed land use in Beijing, land use types should be reduced at the block scale and the area of different land use types should be balanced at the life circle scale.