Exploratory Analysis for Big Social Data Using Deep Network

Abstract

Exploratory analysis is an important way to gain understanding and find unknown relationships from various data sources, especially in the era of big data. Traditional paradigms of social science data analysis follow the steps of feature selection, modeling, and prediction. In this paper, we propose a new paradigm that does not require feature selection so that data can speak for itself without manually picking out features. Besides, we propose using the deep network as a methodology to explore previously unknown relationships and capture complexity and non-linearity between target variables and a large number of input features for big social data. The new paradigm tends to be a relatively generic approach that can be widely used in different scenarios. In order to validate the feasibility of the paradigm, we use country-level indicators forecasting as a case study. The process includes: 1) data collection and preparation and 2) modeling and experiment. The data collection and preparation part builds a data warehouse and conducts the extract-transform-load process to eliminate data format inconsistency. The modeling and experiment part includes model setup and model structures change to achieve relatively high accuracy on prediction results at both model level and case level. We find some patterns about network capacity modification and the influence of time interval difference on the test results, whereas both of them deserve further research.