A New Performance Evaluation Method for Time Series Models

Abstract

In this work, we propose a new time series prediction error calculation technique and model assessment method. We discuss inadequacy of traditional metrics in evaluating success of predictive time series models. One naive but ineffective way of predicting the future is giving out today's realized values as tomorrow's prediction. However, conventional error metrics such as RMSE, MSE and AE etc. seem to give very good results when we employ this kind of a naive approach. Many new models based on machine learning techniques end up mimicking this incompetent behaviour, but earning high marks from conventional metrics. In addition to the performance metrics obtained by classical approaches for the performance of time series, we propose a new metric and model comparison technique, based on the need for an approach which is far from the naive and ineffective approach we have just mentioned. In this study, we calculated the predictive powers of LSTM, GRU, dense, simpleRNN and the hybrid layered architectures that we have created in addition to the standard deep learning architectures such as LSTM, GRU, dense and RNN for estimating stock prices. Among these models, we determined the best model according to our suggested evaluation technique.