Detecting physically unrealistic outliers in ACE-FTS atmospheric measurements

P E Sheese,C D Boone,K A Walker

doi:10.5194/amt-8-741-2015

Abstract

Abstract. The ACE-FTS (Atmospheric Chemistry Experiment – Fourier Transform Spectrometer) instrument on board the Canadian satellite SCISAT has been observing the Earth's limb in solar occultation since its launch in 2003. Since February 2004, high resolution (0.02 cm−1) observations in the spectral region of 750–4400 cm−1 have been used to derive volume mixing ratio profiles of over 30 atmospheric trace species and over 20 atmospheric isotopologues. Although the full ACE-FTS level 2 data set is available to users in the general atmospheric community, until now no quality flags have been assigned to the data. This study describes the two-stage procedure for detecting physically unrealistic outliers within the data set for each retrieved species, which is a fixed procedure across all species. Since the distributions of ACE-FTS data across regions (altitude/latitude/season/local time) tend to be asymmetric and multimodal, the screening process does not make use of the median absolute deviation. It makes use of volume mixing ratio probability density functions, assuming that the data, when sufficiently binned, are at most tri-modal and that these modes can be represented by the superposition of three normal, or log-normal, distributions. Quality flags have been assigned to the data based on retrieval statistical fitting error, the physically unrealistic outliers described in this study, and known instrumental/processing errors. The quality flags defined and discussed in this study are now available for all level 2 versions 2.5 and 3.5 data and will be made available as a standard product for future versions.

Highlights

One of the most common techniques for screening out anomalous data from a data set is to calculate the set’s mean (μ) and standard deviation (σ )
Similar, method is to use the median and MAD (Leys et al, 2013; Toohey et al, 2010, and references therein), in place of the mean and standard deviation respectively, where, MAD = mediani xi − medianj xj. This method is much less sensitive to extreme outliers, as the presence of outliers typically has an insignificant effect on the median value
In the case of data that are multimodal or asymmetrically distributed and contain multiple extreme outliers, it is likely that neither the σ nor the MAD will be an appropriate estimate of the variation, or scale, of the measurements

Summary

Introduction

One of the most common techniques for screening out anomalous data from a data set is to calculate the set’s mean (μ) and standard deviation (σ ). For both subsets of H2O data, inlier limits were determined for μ ±3σ and median ±3 MAD × 1.428 (1.428 is the scale factor for the MAD to equal the σ assuming a normal distribution (Rousseeuw and Croux, 1993)). For ACE-FTS data, a tolerance level, determined empirically, of 0.025 is used, which corresponds to a 97.5 % confidence of an excluded data point being an outlier, i.e. any value x where x Ex dx or x∞E x dx is less than 0.025 is rejected This method, required determining an analytical solution for the data’s EDF.

2.26 O13C18O

Findings

Conclusions