MCSketch: An Accurate Sketch for Heavy Flow Detection and Heavy Flow Frequency Estimation

Abstract

AbstractAccurately finding heavy flows in data streams is challenging owing to limited memory availability. Prior algorithms have focused on accuracy in heavy flow detection but cannot provide the frequency of a heavy flow exactly. In this paper, we designed a two-mode counter, called Matthew Counter, for the efficient use of memory and an accurate record flow frequency. The key ideas in Matthew Counter are the use of idle high-bits in the counter and the adoption of a power-weakening method. Matthew Counter allows sufficient competition during the early stages of identifying heavy flows and amplifying the relative advantage when the counter is sufficiently large to ensure the level of accuracy. We also present an invertible sketch, called MCSketch, for supporting heavy-flow detection with small and static memory based on Matthew Counter. The experiment results show that MCSketch achieves a higher accuracy than existing algorithms for heavy flow detection. Moreover, MCSketch reduces the average relative error by approximately 1 to 3 orders of magnitude in comparison to other state-of-art approaches.KeywordsData stream miningHeavy flow detectionSketch