Linear and sublinear time algorithms for mining frequent traversal path patterns from very large Web logs

Abstract

This paper aims for designing algorithms for the problem of mining frequent traversal path patterns from very large Web logs with best possible efficiency. We devise two algorithms for this problem with the help of fast construction of shallow generalized suffix trees over a very large alphabet. These two algorithms have respectively provable linear time and sublinear complexity, and their performance is analyzed in comparison with the two a priori-like algorithms in (Chen et al., 1998) and the well-known Ukkonen algorithm for online suffix tree construction (1995). It is shown that these two algorithms are substantially efficient than the two apriori-like algorithms and the Ukkonen algorithm. The linear time algorithm has optimal performance in theory, while the sublinear time algorithm has better empirical performance.