(Un)wisdom of Crowds: Accurately Spotting Malicious IP Clusters Using Not-So-Accurate IP Blacklists

Abstract

Most complex tasks on the Internet-both malicious and benign-are collectively carried out by clusters of IP addresses. We demonstrate that it is often possible to discover such clusters by processing data sets and logs collected at various vantage points in the network. Obviously, not all clusters discovered in this way are malicious. Nevertheless, we show that malicious clusters can accurately be distinguished from benign ones by simply using an IP blacklist and without requiring any complex analysis to verify malicious behavior. In this paper, we first propose a novel clustering framework which can be applied on data sets of network interactions to identify IP clusters carrying out a specific task collectively. Then, given such a list of identified clusters of IP addresses, we present a simple procedure to spot the malicious ones using an IP blacklist. We show that by choosing the parameter of the proposed clustering process optimally using a blacklist, hence making it blacklist-aware, we significantly improve our overall ability to detect malicious clusters. Furthermore, we mathematically show that even a blacklist with poor accuracy can be used to detect malicious clusters with high precision and recall. Finally, we demonstrate the efficacy of the proposed scheme using real-world login events captured at the login servers of a large webmail provider with hundreds of millions of active users.