End-to-end LDA-based automatic weak signal detection in web news

Abstract

An extremely competitive business environment requires every company to monitor its competitors and anticipate future opportunities and risks, creating a dire need for competitive intelligence. In response to this need, foresight study became a prominent field, especially the concept of weak signal detection. This research area has been widely studied for its utility, but it is limited by the need of human expert judgments on these signals. Moreover, the increase in the volume of information on the Internet through blogs and web news has made the detection process difficult, which has created a need for automation. Recent studies have attempted topic modeling techniques, specifically latent Dirichlet allocation (LDA), for automating the weak signal detection process; however, these approaches do not cover all parts of the process. In this study, we propose a fully automatic LDA-based weak signal detection method, consisting of two filtering functions: the weakness function aimed at filtering topics, which potentially contains weak signals, and the potential warning function, which helps to extract only early warning signs from the previously filtered topics. We took this approach with a famous daily web news dataset, and we could detect the risk of the COVID19 pandemic at an early stage.