Space-efficient representation of truncated suffix trees, with applications to Markov order estimation

Abstract

Listen

Translate article

Suffix trees (ST) are useful in many text processing applications, for example, to determine the number of occurrences of patterns of arbitrary length in an input string x. If the length n, of x, is large, the memory required to represent the ST may become a practical performance bottleneck. This problem can be alleviated, in cases where a nontrivial upper bound is known on the lengths of the patterns of interest, by using a truncated ST (TST). However, conventional TST implementations still require Ω(n) bits of memory, since they store x. We describe a new TST representation that avoids this limitation by storing all the information necessary to reconstruct the TST edge labels in a string y that is often much shorter than x. We apply TSTs to the implementation of Markov order estimators, where an upper bound kn on the estimated order can be derived or it is imposed (for consistency, for example). The new representation allows for estimator implementations with sublinear space complexity in some cases of interest. In other cases we show, experimentally, that even when the new representation does not have an asymptotic advantage, it still achieves very significant memory savings in practice.