Mining Distributed Frequent Itemsets Using a Gossip Based Protocol

Abstract

Recently, there has been a growing attention in frequent item set mining in distributed systems. In this paper, we present an algorithm to extract frequent item sets from large distributed datasets. Our algorithm uses gossip as the communication mechanism and does not rely on any central node. In gossip based communication, nodes repeatedly select other random nodes in the system, and exchange information with them. Our algorithm proceeds in rounds and provides all nodes with the required support counts of item sets, such that each node is able to extract the global frequent item sets. For local iteration and generation of candidate item sets, a trie data structure is used, which facilitates the process and reduces execution time. We further propose an improvement to our algorithm by grouping nodes and arranging them into a hierarchical structure. By performing aggregation tasks in groups, communication overhead is effectively reduced. We evaluate our proposal using simulation, and show advantages of our algorithms in reducing execution time and communication overhead, while preserving accuracy.