Online Percentile Estimation (OPE)

Abstract

We describe a control loop over a sequential online algorithm. The control loop either computes or uses the sequential algorithm to estimate the temporal percentiles of any univariate input data sequence. It transforms an input sequence of numbers into output sequence of histograms. This is performed without storing or sorting all of the observations. The algorithm continuously tests whether the input data is stationary, and reacts to events that do not appear to be stationary. It also indicates how to interpret the histograms since their information content varies according to whether a histogram was computed or estimated. It works with parameter-defined, fixed-size small memory and limited CPU power. It can be used for a statistical reduction of a numerical data stream and, therefore, applied to various Internet of Things applications, edge analytics or fog computing. The algorithm has a built-in feasibility metric called applicability. The applicability metric indicates whether the use of the algorithm is justified for a data source: it works for an arbitrary univariate numerical input, but it is statistically feasible only under some requirements, which are explicitly stated here. We also show the results of a performance study, which was executed on the algorithm with synthetic data.