Data Analysis and Crawler Application Implementation Based on Python

Abstract

In this age of information explosion, how to find the data we want efficiently from various miscellaneous data and extract them from the network in batches has become a key problem. And sometimes the data not processed itself may be confusing for people, through what kind of technical means, how to get to the complex data processing, finally become a kind of intuitive figures, or the trend that people can directly extract information from is also a very important topic to study in the era of the data. This topic will choose Steam online game platform as the research object. Steam is an online game retail platform launched by Valve company in the United States in 2003. Under the real circumstance, to explore how to develop a complete crawler method based on Scrapy frameworks for Steam top sell list and publishers, developers and stores page, to crawl the various data of all works of developers and publishers under the page on Steam platform. Based on the crawled data, use basic data analysis to analysis user's favorite game types in the top sell list, the total number of releases of game platforms of certain developers and publishers, the proportion of favorable comments, etc. and extract useful information through the data analysis process. To draw the conclusion and make summary in the final. In short, at first, this paper will explore how to develope a crawler with scontrolable and automatic crawling abilitys which can crawl specific target; Then the data that is crawled will be analysised and visualized by using Pandas library and Matplotlib library, the useful information will be extracted from the data analysis and visualization process, so as to complete.