Multiset Membership Lookup in Large Datasets

Abstract

Given a dataset <inline-formula><tex-math notation="LaTeX">$\mathcal S$</tex-math></inline-formula> composed of <inline-formula><tex-math notation="LaTeX">$g$</tex-math></inline-formula> subsets with each data item belonging to one of them, <i>multiset membership lookup</i> takes an item <inline-formula><tex-math notation="LaTeX">$e$</tex-math></inline-formula> as input and outputs a binary answer whether <inline-formula><tex-math notation="LaTeX">$e\in {\mathcal S}$</tex-math></inline-formula> and, in case of yes, the ID of the subset to which <inline-formula><tex-math notation="LaTeX">$e$</tex-math></inline-formula> belongs. Overlaid upon while more sophisticated than the canonical membership lookup, multiset membership lookup emerges as a pivotal functionality in many computing and networking paradigms. The quest to achieve high-speed, high-accuracy lookup with limited memory cost makes lookup algorithm design a challenging task, particularly when the data items arrive as a stream. In this paper, we devise compact data structures and lookup algorithms that are amendable for hardware implementation, while guaranteeing high lookup accuracy and supporting interactive query processing. We first propose <i>multi-hash color table</i> , a variant of Bloom filter, to encode subset IDs compactly and map the ID of an item to its subset ID. We further construct a more balanced data structure called <i>balanced multi-hash color table</i> to improve the compactness by integrating the state-of-the-art load balancing technique. We complete our work by addressing the case of <i>batch arrivals</i> and design a batched recording algorithm optimizing the memory efficiency. We give both theoretical and empirical analysis to characterize and evaluate the performance of the proposed algorithms in terms of lookup accuracy, memory and access efficiency.