An improved algorithm for cleaning Ultra High-Frequency data

Abstract

We develop a multiple-stage algorithm for detecting outliers in Ultra High-Frequency financial market data. We show that an efficient data filter needs to address four effects: the minimum tick size, the price level, the volatility of prices and the distribution of returns. We argue that previous studies tend to address only the distribution of returns, and may tend to ‘overscrub’ a data set. In this study, we address these issues in the market microstructure element of the algorithm. In the statistical element, we implement the robust median absolute deviation method to take into account the statistical properties of financial time series. The data filter is then tested against previous data-cleaning techniques and validated using a rich individual equity options transactions data set from the London International Financial Futures and Options Exchange. The paper has many relevant insights for any practitioner who uses high frequency derivatives data, for example, for market analysis or for developing trading strategies.