VATE: A trade-off between memory and preserving time for high accurate cardinality estimation under sliding time window

Abstract

Host cardinality is one of the most important attributes in the field of network research. The cardinality estimation under sliding time window has become a research hotspot in recent years. This kind of algorithms preserve the time information of sliding time window by introducing more powerful counters. The more counters used in these algorithms, the higher the estimation accuracy of these algorithms will be. However, the available number of sliding counters is limited due to their large memory footprint or long state-preserving time. To solve this problem, a new sliding counter, asynchronous time stamp (AT), is designed in this paper. AT has the advantages of small memory consumption and low state-preserving time. It can directly replace the counter used in the existing algorithms. On the same platform, higher accuracy can be achieved by adopting more AT. Furthermore, this paper designs a new per host cardinality estimation algorithm, virtual AT estimator (VATE), based on AT. VATE is also a parallel algorithm that can be deployed on GPU. With the parallel processing capability of GPU, VATE can estimate cardinalities of hosts in a 40 Gb/s high-speed network in real time at the time granularity of 1 s. In our experiments, VATE increases the state-preserving speed by 4 to 400 times at the cost of 11.11% more memory compared with a state-of-the-art algorithm.