A permutation approach to evaluating the performance of a forecasting model of methane content in the atmospheric surface layer of arctic region

Abstract

The time series forecasting models of climate and ecological systems have grown in number, variety and complexity over the past years. In our work, we proposed to use the ideology of testing statistical hypotheses to evaluate the performance of a predictive time series model. The data for this study were obtained during July–August 2017 from the surface concentration dynamics monitoring of the main greenhouse gases at the Arctic Island of Belyy, Yamalo-Nenets Autonomous Okrug, Russia. Models based on the two types of artificial neural networks - simple recurrent Elman network and nonlinear autoregressive model with exogenous input (NARX), predicted changes in methane concentration. Totally, six models were considered: Elman, NARX and their hybrid model using Daubechies wavelet (db4) and Symlet wavelet (sym4) transformed data for training. The performance of the models was evaluated using 12 ″traditional” indices (correlation and coefficient of determination, Wilmott agreement indices, the mean absolute and mean square errors, etc.) and the proposed permutation approach using the ideology of testing statistical hypotheses. In general, on the data under study, the estimates of the permutational approach and other accuracy indicators and error are the same, but there are some discrepancies. Effect sizes do not always determine the statistical significance of differences. Different artificial neural networks forecasting models are tuned to capture different patterns in the time series of greenhouse gases in the atmosphere. The permutational approach is useful in determining the best model for prediction. So, to predict the variability of methane content, the hybrid model autoregressive neural networks NARX (sym4) is the best performing (means difference = −0.00373 ppm; p-value = 0.01327; correlation coefficient = 0.83320; p-value = 0.00000), and to predict the average concentration of methane content for the entire forecast period, the recurrent neural network Elman (sym4) model is the best (means difference = 0.00241 ppm; p-value = 0.17758; correlation coefficient = 0.50388; p-value = 0.01986).