Some Comments on the Analysis of “Big” Scientific Time Series

Abstract

Experience with long time series from space, climate, seismology, and engineering has demonstrated the need for even longer data series with better precision, timing, and larger instrument arrays. We find that almost all the data we have examined, including atmospheric, seismic data, and dropped calls in cellular phone networks contain evidence for solar mode oscillations that couple into Earth systems through magnetic fields, and that these are often the strongest signals present. We show two examples suggesting that robustness has been overused and that many of the extremes in geomagnetic and space physics data may be the result of a superposition of numerous modes. We also present initial evidence that the evolution of turbulence in interplanetary space may be controlled by modes. Returning to the theme of “big data,” our experience has been that theoretical predictions that spectra would be asymptotically unbiased have turned out to be largely irrelevant with very long time series primarily showing that we simply did not understand the problems. Data that were considered to have excessively variable spectra appear to evolve into processes with dense sets of modes. In short data blocks, these modes are not resolved and as the relative phase of the modes within the estimator varies, so does the apparent power. Ideas that data series become uncorrelated at modest distances in either time or space do not seem to be true with the long duration continuous time series data we have examined.