Robust Estimation With Sampling and Approximate Pre-Aggregation

Abstract

This chapter considers the problem of approximation of aggregate functions over categorical data, or mixed categorical/numerical data. It proposes a method based upon random sampling, called approximate pre-aggregation (APA), a framework for using simple summary statistics to greatly increase the accuracy of random sampling for estimation of aggregate queries over categorical or mixed categorical/numerical data. This is important because many previous estimation techniques have largely ignored categorical data. APA is based upon sound, statistical techniques such as maximum likelihood estimation and constrained quadratic programming. It is also suitable for estimation in a streaming environment, since the information used by APA can be collected in a single database scan. The biggest drawback of sampling for aggregate function estimating is the sensitivity of sampling to attribute value skew, and APA uses several techniques to overcome this sensitivity. The increase in accuracy using APA compared to “plain vanilla” sampling is dramatic. For SUM and AVG queries, the relative error for random sampling alone is more than 700% greater than for sampling with APA. Even if stratified sampling techniques are used, the error is still between 28% and 175% greater than for APA.