Improving the prediction of complex nonlinear turbulent dynamical systems using nonlinear filter, smoother and backward sampling techniques

Abstract

Predicting complex nonlinear turbulent dynamical systems using partial observations is an important topic. Despite the simplicity of the forecast based on the ensemble mean time series, several critical shortcomings in the ensemble mean forecast and using path-wise measurements to quantify the prediction error are illustrated in this article. Then, a new ensemble method is developed for improving the long-range forecast. This new approach utilizes a mixture of the posterior distributions from data assimilation and is more skillful in predicting non-Gaussian statistics and extreme events than the traditional method by simply running the forecast model forward. Next, a systematic framework of improving forecast models is established, aiming at advancing the predictions at all ranges. The starting model in this new framework belongs to a rich class of nonlinear systems with conditional Gaussian structures. These models allow an efficient nonlinear smoother for state estimation using partial observations, which in turn facilitates a rapid parameter estimation based on an expectation–maximization algorithm. Conditioned on the partially observed time series, the nonlinear smoother further advances an efficient backward sampling of the hidden trajectories, the dynamical and statistical characteristics from which allow a systematic quantification of model error through information theory. The sampled trajectories then serve as the recovered observations of the hidden variables that promote the use of general nonlinear data-driven modeling techniques for a further improvement of the forecast model. A low-order model of the layered topographic equations with regime switching and rare events is used as a test example to illustrate this framework.