Burst Hotspots Dynamic Detection and Tracking on Large-Scale Text Stream

Abstract

The online social network has become an important communication tool for people and forms a virtual society interacting with the real world. The numerous events rapidly spread through social networks and may become hotspots in a short period of time. Especially, the negative events vibrate national security and social stability, potentially causing a series of social problems. Therefore, detection and tracking burst hotspots on social networks are of great significance. However, this problem is non-trivial because of the challenges of massive noise, sparsity, high-dimensionality, and dynamic changing. To address these challenges, this paper proposes a distributed method of burst hotspots dynamic detection and tracking based on Map/Reduce. To judge the relevance of the current text and previous texts, the keyword similarity matrix is calculated by using Word2Vec and context information. The keyword weights are modified according to the association model to further reduce noise. The proposed method is not only robust to the sparsity of short text but also overcomes the curse of dimensionality. Finally, it can dynamically detect and track burst hotspots from large-scale short text stream on the Hadoop platform. The experiments have shown that the proposed method outperforms state-of-the-art algorithms.