The distribution of empirical periodograms: Lomb-Scargle and PDM spectra

Abstract

The theoretical probability distributions of periodograms are derived for the assumed variance of noise. In practice, however, the variance is estimated from data and hence it is a random variable itself. The empirical periodograms, i.e. the periodograms normalized using the estimated variance, therefore follow a distribution different from that predicted by theory. We demonstrate that in general many empirical periodograms follow the beta distribution. In particular, as an example we consider a Lomb &38; Scargle (L-S) modified power spectrum with an exponential theoretical distribution. We derive its easy-to-use analytical empirical distribution. We demonstrate that the difference between the tails of the empirical and theoretical distributions is large enough to have a profound effect on the statistical significance of signal detections. The difference persists despite generally good asymptotic convergence of the distributions near their centres. Hence we argue that even for well-behaved statistics (e.g. L-S) one has to use our new empirical beta distributions rather than the theoretical ones. Our conclusions are illustrated by a realistic example. In the example we demonstrate a significant difference between the theoretical and empirical distributions. Additionally, we provide an example of conversion between analysis of variance (AOV), power-spectrum, PDM and χ2 periodograms.