Crawler Program Research Articles

LDA-based Topic Mining of Microblog Comments

Microblog comments express the public’s views and attitudes on Microblog hot topics and hot events. Therefore, by excavating the theme of Microblog comments, we can help people understand the trend of public opinion and the attitude of the public towards the event. In view of Microblog comment topic mining, the problem we need to solve is how to use simple methods to quickly and efficiently obtain Microblog comment content, and how to use what methods to mine the theme of Microblog hot events to a great extent. First of all, we construct dynamic links through static web pages, write Microblog comment web crawler program in Python language, and solve the problem of crawling Microblog dynamically loading web page comments. Secondly, we use the Latent Dirichlet Allocation model, which is the implicit Dirichlet distribution, to mine the text.

Exploring E-Commerce Big Data and Customer-Perceived Value: An Empirical Study on Chinese Online Customers

The purpose of this study is to make good use of the massive amount of online user comment data to explore and analyze the dimensions of customer-perceived value and the importance of each dimension, given the background of China’s huge e-commerce market. We compiled a web crawler program to collect online comment data from online reviews. The crawled data were pre-processed and content analysis were performed. A customer-perceived value dictionary was constructed based on the extraction of frequent terms, literature review, and expert opinions. We re-identified the dimensions of customer-perceived value to include four key dimensions and corresponding subdivisions. Both the rationality and operability of the dimension model of customer-perceived value were validated and applied. Thereafter, the importance of various dimensions and the impacts of customer-perceived value dimensions on customer loyalty were analyzed and discussed. The empirical research results reveal that all four dimensions of customer-perceived value play an important role in customer-perceived value and that the patterns and degrees of the role of each dimension are rather different. Further, only certain parts of the dimensions of customer-perceived value have an impact on customer loyalty, and the degree of the impact differs substantially.

IJMLNCE Editorial Note Volume No 04, Issue No 01

IJMLNCE Editorial Note Volume No 04, Issue No 01

CLMIP: cross-layer manifold invariance based pruning method of deep convolutional neural network for real-time road type recognition

Recently, deep learning based models have demonstrated the superiority in a variety of visual tasks like object detection and instance segmentation. In practical applications, deploying advanced networks into real-time applications such as autonomous driving is still challenging due to expensive computational cost and memory footprint. In this paper, to reduce the size of deep convolutional neural network (CNN) and accelerate its reasoning, we propose a cross-layer manifold invariance based pruning method named CLMIP for network compression to help it complete real-time road type recognition in low-cost vision system. Manifolds are higher-dimensional analogues of curves and surfaces, which can be self-organized to reflect the data distribution and characterize the relationship between data. Therefore, we hope to guarantee the generalization ability of deep CNN by maintaining the consistency of the data manifolds of each layer in the network, and then remove the parameters with less influence on the manifold structure. Therefore, CLMIP can be regarded as a tool to further investigate the dependence of model structure on network optimization and generalization. To the best of our knowledge, this is the first time to prune deep CNN based on the invariance of data manifolds. During experimental process, we use the python based keyword crawler program to collect 102 first-view videos of car cameras, including 137 200 images (320 × 240) of four road scenes (urban road, off-road, trunk road and motorway). Finally, the classification results have demonstrated that CLMIP can achieve state-of-the-art performance with a speed of 26 FPS on NVIDIA Jetson Nano.

Financial Hedging and Corporate Cash Policy

We study the implications of financial hedging for corporate cash policy and the value of cash holdings. Using a web crawler program to collect data on the use of financial derivatives between 1993 and 2016, we find that US public firms with financial hedging programs have smaller cash reserves but a higher value of cash than firms without hedging contracts in place. Our empirical results are robust when controlling for potential endogeneity issues, corporate governance, cash regimes, and alternative measures of cash holdings. Further, we find that financial hedging not only increases the investment sensitivity to internal cash, but also has a positive effect on investment efficiency. The positive effect of financial hedging on the value of cash is more pronounced for firms with more financial constraints, higher information asymmetry, and weaker corporate governance. Collectively, our paper highlights the importance of corporate cash policy as a channel through which financial risk management increases firm value.

What Causes Different Sentiment Classification on Social Network Services? Evidence from Weibo with Genetically Modified Food in China

(1) Background Genetic Modification (GM) refers to the transfer of genes with known functional traits into the target organism, and ultimately the acquisition of individuals with specific genetic traits. GM technology in China has developed rapidly. However, the process is controversial; thus, future development may be hindered. China has become the world’s largest importer of GM products. Research on the attitudes towards GM food in China will help the government achieve sustainable development by better understanding and applications of the technology. (2) Methods This research utilizes data from Sina Weibo (microblog), one of the biggest social network services (SNS) in China. By using the self-created Python crawler program, comments related to the genetically modified food in the People’s Daily account are analyzed. Sentiment classifications are analyzed via multivariate logistic regression. (3) Results Based on the factor analysis, theme type characteristics, the propagation characteristics, the body information characteristics, and the comment characteristics have different degrees of influence on the user’s emotional distribution. (4) Conclusion Practical implications and conclusions are provided based on the results at the end.

Big Data Analysis Applied in Agricultural Planting Layout Optimization

Abstract. Mile is a region in Yunnan Province, China. The planting-related industry is its pillar industry. Its agricultural population accounts for 59.3% of the total population. Temporal fluctuations of crop price and yield have a significant influence on farmers’ revenue. Farmers’ selection of crop species, crop planting strategy, and agricultural planting layout according to the market price is important in securing their revenue. In this study, we used a web crawler program to obtain a large amount of data on agricultural product prices from the Internet. Then, the price fluctuation trend of the main economic crops was analyzed by using the K-means clustering method. The net investment yield and the Sharpe ratio were used to compare the economic benefits and investment risks of 10 crops and five cultivation strategies in Mile. Furthermore, a comprehensive comparative advantage index, which integrates the net investment yield, Sharpe ratio, scale advantage index, productivity advantage index, and ecological suitability advantage index, was adopted to comprehensively measure the advantages of crop cultivation. Finally, we propose a spatial-temporal big data analysis model based on the cuckoo search algorithm to optimize the spatial layout of the main crops in Mile in 2017. Based on the comparative analysis of the remote sensing monitoring results and the spatial optimization layout results of the main crops in 2017, several suggestions were given. The results based on agricultural big data analysis, such as crop selection cluster analysis, economic benefit analysis, and crop planting layout optimization, can give suggests to farmers plant suitable crops on right lands, in right time. Thus, it can help farmers stabilize their revenue and minimize the risk by choosing the right crops and planting strategy in accordance with the local conditions. Keywords: Agriculture investment risk, Agricultural layout optimization, Cuckoo search algorithm, K-means clustering, Relative advantage analysis, Spatial-temporal big data analysis.

Design of Data Capture Program Based on Web Crawler Technology

With the advent of the Internet of things era and the vigorous development of electronic information era, the network information resources are growing exponentially. Faced with the demand of obtaining useful information, based on the general web crawler technology, this paper uses Python software to design a deep and optimized web crawler data fetching program. This crawler program can effectively solve a series of problems, such as waiting time, information overlapping and information incompleteness, so that the crawler has good performance.

Urban Spatial Interaction Analysis Using Inter-City Transport Big Data: A Case Study of the Yangtze River Delta Urban Agglomeration of China

A better understanding of the urban spatial interaction is important for optimizing the spatial structure and layout of urban agglomeration (UA). We develop a crawler program to compile online big data for urban spatial interaction analysis. Differing from the previous studies, vectorial, realistic, and high spatiotemporal resolution inter-city, bus-passenger-flow big data instead of statistical and modeled data are used for urban spatial interaction analysis. The Yangtze River Delta (YRD) is selected as a case study region to test the big data approach and to gain insights into the cities’ interaction in China’s largest UA. The results testified the superiorities of the big-data method over the traditional gravity model, confirmed some phenomena discussed or mentioned in the literature and regional plans regarding the urban interaction in YRD, derived policy implications for enhancing the sustainability of UA, and suggested some potentials for improving the limitations of the big-data method.

Method for division of urban load power supply district based on cluster analysis

The division of urban power supply district is an essential issue in the medium voltage distribution network plan. This study develops a method for division of an urban load power supply district, integrating the open source data into the distribution network planning, at first, raw georeferenced point of information data is crawled by crawler program based on location retrieval service interface (Place API), the buildings' data of the planned urban district is extracted and power load estimation are introduced in detail, the dataset of the low-voltage load spatial distribution is set-up; secondly, the clustering algorithm selects both the local density of samples ρ i and the distance between samples δ i as criteria to form the clusters, cluster centres are recognised from the binary pair in `decision map', as load density peaks. Thirdly, the spatial distribution dataset of the low-voltage users is taken as the data points in the clustering algorithm; the result of clusters corresponds to division in the power distribution with certain capability. Consequently, the methodologies proposed are verified on one example district of ~71.7587 hectares, the division scheme can provide theoretical guidance for the location and sizing of power distributors in the urban distribution network.

The Study of Using VGI to Analyze the Tourist Satisfaction About Taichung Jazz Festival

Since 2003, the Taichung Jazz Festival has become one of the major annual events regularly held in Taichung City. The number of tourists and the tourism business opportunities brought by this festival has been increasing year by year, even reaching more than 1 million participating tourists/times ever since 2015. In terms of traditional assessment methods for great events, we used to obtain analytic such as visitor satisfaction or the number of people through questionnaires. However, different levels of issues concerned by tourists cannot be easily understood through standardized questionnaires. Due to the popularization of online platforms and smart phones, people tend to voluntarily provide some when they are participating in an activity. Such coordinated is namely Volunteered geographic information (VGI), ex. check-in created by anyone. People can show their positive and negative messages by expressing their words about certain places (food, landscape, etc.), which can can make up for the shortcomings of traditional questionnaires. In this study, through the API provided by Facebook and by writing a web crawler program, we downloaded a total of 46,260 comments/messages written by people during the period of the Jazz Festival. Then, by means of Chinese word segmentation and through keywords, statistical analyses were conducted on two indicators shown by these tourists regarding the Jazz Festival: 1. Satisfaction about this event: To analyze people's positive and negative evaluations of the handling of this event, as well as their feelings; 2. Suggestions for event improvements: To analyze all aspects of concrete problems and suggestions for improvements proposed by people for this event. In this study, through collecting VGI data and constructing unstructured analysis methods, explorations were made, concerning people's intuitive feeling about Jazz Festival from a mass perspective. In addition, comparisons and analyses against traditional questionnaires were conducted. Therefore, the findings of this study can serve as a reference for future leisure activity surveys combined with VGI data analyses.

In sharing economy we trust: the effects of host attributes on short-term rental purchases

PurposeTrust has been widely recognized as the crucial factor of consumer purchase intention when shopping on peer-to-peer short-term rental platforms where hosts and renters are strangers. However, the specific attributes of hosts that help build trust with potential renters and drive their purchase of short-term rentals remain unknown. This study aims to explore the effects of host attributes on renter purchases made on Xiaozhu.com, one of the top short-term rental platforms in China, while controlling for short-term rental characteristics.Design/methodology/approachA crawler program was developed by Python to collect the host attributes and their short-term rental characteristics of 935 hosts in Beijing from November 18, 2015 to February 14, 2016. The authors use Poisson regression models to estimate the effects of host attributes on renter reservations. They also conduct a series of robustness checks for the estimated results.FindingsThe authors found that host attributes such as the time of reservation confirmation, the acceptance rate of renter reservations, the number of listings owned, whether a personal profile page is disclosed and gender of the host significantly affect renter reservations, whereas the response rate of the host does not influence renters when purchasing short-term rentals online.Originality/valueThis study identifies which host attributes are perceived as trustworthy and affect renters’ purchase decisions, a topic of both theoretical and practical importance but currently less researched. The findings add to emerging literature by providing insights on trust-building in the peer-to-peer economy. Useful suggestions are also provided on strengthening the trust mechanism on short-term rental platforms to facilitate peer-to-peer transactions. Notably, the study is the first attempt to examine the perception of Chinese users toward short-term rentals despite its global prevalence. The analytical insights revealed from large scale but granular online observations data of host attributes and actual renter reservations greatly supplement findings of extant literature using survey and experiment approaches.

Registry Metadata Quality Assessment by the Example of re3data.org Schema

Nowadays, research data repositories (RDR) have become progressively widespread all over the world. To expand repository services and build up inbound linking strategy, organizations list their repositories with so called Global Registries. Accordingly, such registries should be carefully described by the related data. In this study, we explore the metadata schema of re3data.org . We collect and analyze descriptions from the listed repositories, and come up with some suggestions concerning possible improvements to the metadata schema. To accomplish this, we develop a crawler program, which collects necessary data from the re3data.org . Based on the analysis results, we have identified two issues that required elements is missing, one issue that required element value is missing when the corresponding property is applied, five inconsistency issues with re3data controlled vocabulary, six issues with undescribed optional elements, and two inconsistency issues between the elements and their attributes which do not pair with. We believe this discussion can facilitate improvements to the existing re3data.org schema and further help researchers who analyze data repository trends. URL: http://ijkcdt.net/xml/10971/10971.pdf

Car Sales Analysis Based on the Application of Big Data

Whether industrial 4.0 nor Internet industry, for today's industrial manufacturing enterprises, it should be to make full use of information and communication technology to deal with the arrival of smart and effective large data, combining products, machinery and human resources into together, according to the unexpected speed about the mode of sales product, it can change the manufacturing enterprises to process innovation and reform. This paper takes the automobile manufacturing industry as an example, based on sale car large data analysis, using data mining technology, through the Java program to prepare web crawler program for data collection. To give some suggestions for the automobile manufacturing industry in the production of automobile, it reduces the inventory of automobile enterprises and the waste of resources.

Longitudinal analysis of online community dynamics

Purpose– The purpose of this paper is to explore the dynamics of an online community by examining its participants’ centrality measures: degree, closeness, and the betweenness centrality. Each centrality measure shows the different roles and positions of an individual participant within an online community. To be specific, this research examines how an individual participant’s role and position affects her/his information sharing activities within an online community over time. Additionally, it investigates the differences between two different online communities (a personal interest focussed community and a social interest focussed community), in terms of the interaction patterns of participants.Design/methodology/approach– For this research, the authors collected log files from Korean online discussion communities (café.naver.com) using a crawler program. A social network analysis was used to explore the interaction patterns of participants and calculate the centrality measures of individual participants. Time series cross-sectional analysis was used to analyze the effects of the roles and the positions on their information sharing activities in a longitudinal setting.Findings– The results of this research showed that all three centrality measures of an individual participant in previous time periods positively influenced his/her information sharing activity in the current periods. In addition, this research found that, depending on the nature of the discussion issues, the participants showed different interaction patterns. Throughout this research, the authors explored the interaction patterns of individual participants by using a network variable, the centrality, within a large online community, and found that the interaction patterns provided strong impact on their information sharing activities in the following months.Research limitations/implications– To investigate the changes of participant’s behaviors, this study simply relies on the numbers of comments received and posted without considering the contents of the comments. Future studies might need to analyze the contents of the comments exchanged between participants, as well as the social network among participants.Practical implications– Online communities have developed to take a more active role in inviting public opinions and promoting discussion about various socio-economic issues. Governments and companies need to understand the dynamics which are created by the interactions among many participants. This study offers them a framework for analyzing the dynamics of large online communities. Furthermore, it helps them to respond to online communities in the right way and in the right time.Social implications– Online communities do not merely function as a platform for the free exchange and sharing of personal information and knowledge, but also as a social network that exerts massive influence in various parts of society including politics, economy, and culture. Now online communities become playing an important role in our society. By examining communication or interaction behaviors of individual participants, this study tries to understand how the online communities are evolved over time.Originality/value– In the area of online communities, many previous studies have relied on the subjective data, like participant’s perception data, in a particular time by using survey or interview. However, this study explores the dynamics of online communities by analyzing the vast amount of data accumulated in online communities.