Stream-based live public opinion monitoring approach with adaptive probabilistic topic model

Abstract

Public opinion monitoring, also known as first story detection, is defined within the topic detection and tracking on a particular Internet news event. Generally, it is used to find news propagation. Traditional method adopts text matching to address opinion monitoring. But it has some limitations such as hidden and latent topic discovery and incorrect relevance ranking of matching results on large-scale data. In this paper, we propose three solutions to live public opinion monitoring: simple keyword computing and matching, simple probabilistic topic computing and matching, and stream-based live probabilistic topic computing and matching. We point out the disadvantages of the first two solutions such as semantic matching and low efficiency on timely big data. Stream-based real-time topic computing and topic matching with query-time document and field boosting are proposed to make substantial improvements. Finally, our topic computing and matching experiments with crawled historical Netease news records show that our approaches are effective and efficient.