Differentiable Neural Architecture Search for High-Dimensional Time Series Forecasting

Renlong Jie,Junbin Gao

doi:10.1109/access.2021.3055555

Renlong Jie, Junbin Gao

Open Access

https://doi.org/10.1109/access.2021.3055555

Copy DOI

Abstract

This study applies neural architecture search (NAS) techniques to the modeling of high-dimensional time series data such as multi-variate stock indices. It is known that traditional NAS method applies fully connected directed acyclic graph (DAG) for searching cell structures that requires high computational cost and cannot include the two-input operations such as the Hadamard product. To address the drawback of the DAG backbone, a novel two-input backbone cell architecture for recurrent neural networks is proposed, in which each candidate operation is also carried out with two inputs. Instead of using DAG, we simplify the backbone by considering the prior knowledge as an effective backbone such as preserving identity mappings. The cell structures will be incorporated in different types of model architectures including stacked long short-term memory (LSTM), gated recurrent unit (GRU) and attention-based encoder-decoder models. The experimental results on BRICS, G7 and G20 indices indicate that models with recurrent neural network (RNN) cells searched by the proposed backbone structure can significantly outperform baseline models including autoregressive integrated moving average model (ARIMA), vector autoregression (VAR), and stacked LSTM/GRUs. For neural architecture search, the proposed backbone is shown to be more effective compared to the classic differentiable architecture search (DARTS) in both uni-variate and multi-variate time series prediction tasks. Further analysis demonstrates that the pruned cells of the proposed backbone usually contains the Hadamard product introduced as a two-input operation, while the number of parameters involved in these pruned cells is on the same order with the baseline cells.

Highlights

Neural architecture search (NAS) enables learning model structures of artificial neural networks [1]
EXPERIMENTAL RESULTS The experiments will compare the 1-step ahead prediction performance of the following methods: (a) Baseline methods such as Random walk and autoregressive integrated moving average model (ARIMA); (b) long short-term memory (LSTM) model; (c) gated recurrent unit (GRU) model; (d) model with recurrent neural network (RNN) cells searched with differentiable architecture search (DARTS) backbone; and (e) model with RNN cell searched with the proposed RNN backbone
Initial experiments on multi-variate time series forecasting indicate that the newly proposed two-input backbone has the potential to achieve similar or even stronger performance than the traditional LSTM/GRU-based models for multi-variate time series forecasting. It consistently outperform models with RNN cells searched by the DARTS backbone

Summary

Introduction

Neural architecture search (NAS) enables learning model structures of artificial neural networks [1]. Neural network architectures were considered to be discrete hyper-parameters that can be encoded for sequences of discrete variables representing the combinations of neurons, cells and connections. The encoded vectors with corresponding model performances can be handled and optimized by population-based methods such as random search and evolutionary algorithms [2]–[6]. The traditional approach of population-based methods can be computationally expensive for NAS that involves a large number of iterations of full model training for exploration. It is necessary to fully exploit the results of previous trials in order to increase the efficiency of the search process. The associate editor coordinating the review of this manuscript and approving it for publication was Ravinesh C.

Objectives

Methods

Results

Discussion

Conclusion