Discovering Emerging Threats in the Hacker Community: A Nonparametric Emerging Topic Detection Framework

Abstract

The prevalence and rapid growth of cybercrime are largely attributed to hacker communities on the dark web, where cybercriminals extensively exchange hacking resources, share hacking knowledge, and organize cyberattacks. Such streams of hacker-generated content constitute an invaluable data source for developing threat intelligence that can inform organizations of cybersecurity risks and facilitate proactive cyber defense. Drawing upon the design science paradigm, we propose a novel nonparametric emerging topic detection (NPETD) framework for detecting emerging topics in streams of hacker-generated content. Our framework extends the state-of-the-art nonparametric topic model to inductively model topics without having to specify the number of topics a priori. Moreover, our framework features an efficient algorithm to jointly infer topics and detect topic emergence. We conducted experiments to rigorously evaluate the effectiveness and efficiency of our framework in comparison with the state-of-the-art baseline methods. Our framework outperformed the baseline methods in detecting the listings of emerging threats in darknet marketplaces on recall, F-measure, topic coherence, and processor time. The practical utility of our framework is further demonstrated in a major hacker forum, where we identified several notable emerging topics with important implications for victim companies and law enforcement. The proposed framework contributes to cybersecurity, topic detection and tracking, and design science.