OPO: Online public opinion analysis system over text streams

Abstract

With the development of internet, users can express their attitudes towards public events on the social media. Monitoring and analyzing the public opinion can provide effective support for government's policy making. In this paper, a novel online public opinion (OPO) analysis platform over multi-source text streams is proposed. This OPO platform contains three layers: data collection layer, data process and analysis layer, application layer. Here, the network public opinion data is acquired by the popular Selenium component, which can dynamically generate content and parse web pages. Specifically, the platform automatically conducts data processing, including filtering texts to remove the spam information and duplicated content based on shingle algorithm, ranking the content with BM25 algorithm, and recommending high quality relevant data for analysis. The platform adopts hot topic detection and extraction technology, called “cluto”, to capture public opinion. Through these processes, OPO can explicitly analyze public opinion and reveal the focus of public on the hot topics. Experiments on social media stream show that OPO outperforms in public opinion analysis and hot topics extraction.