Asymptotically Optimal Filters

Abstract

A filter, such as a Bloom filter, maintains a compact, probabilistic representation of a set S of elements from a universe U. The challenge is to design a filter that uses optimal space and requires a constant number of memory accesses per query while supporting both insertions and deletions. The two main modern constructions of filters are cuckoo filters and adaptive filters built on quotient filters. Cuckoo filters (Fan et al. 2014) perform well in practice but rely on unproven heuristics. Adaptive filters (Bender et al. 2012, 2018) as well as cuckoo filters experience performance degradation as the number of stored elements approaches the capacity of the filter. Thus, space utilization is sub-optimal. We present a filter that, with high probability, asymptotically achieves optimal space and performance even if the number of elements reaches the capacity of the filter. This construction copes with the problem of false deletes (that it has in common with cuckoo filters) via an elegant connection to adaptive filters. Time permitting, applications that benefit from adaptive filters will be discussed. Joint work with Michael A. Bender, Ioana O. Bercea, Alex Conway, Tomer Even, Martin Farach-Colton, Rob Johnson.