Multiplex visibility graphs to investigate recurrent neural network dynamics

Lorenzo Livi,Cesare Alippi,Robert Jenssen,Filippo Maria Bianchi

doi:10.1038/srep44037

Abstract

A recurrent neural network (RNN) is a universal approximator of dynamical systems, whose performance often depends on sensitive hyperparameters. Tuning them properly may be difficult and, typically, based on a trial-and-error approach. In this work, we adopt a graph-based framework to interpret and characterize internal dynamics of a class of RNNs called echo state networks (ESNs). We design principled unsupervised methods to derive hyperparameters configurations yielding maximal ESN performance, expressed in terms of prediction error and memory capacity. In particular, we propose to model time series generated by each neuron activations with a horizontal visibility graph, whose topological properties have been shown to be related to the underlying system dynamics. Successively, horizontal visibility graphs associated with all neurons become layers of a larger structure called a multiplex. We show that topological properties of such a multiplex reflect important features of ESN dynamics that can be used to guide the tuning of its hyperparamers. Results obtained on several benchmarks and a real-world dataset of telephone call data records show the effectiveness of the proposed methods.

Highlights

Populations of spiking neurons[25]
In this paper we introduce a weighted horizontal visibility graphs (HVGs), with edge values defined as A[i, j]= 1/ (j − i)2 + (x[i] − x[j])2 ∈ [0, 1], ∀ 1 ≤ i, j ≤ tmax
We show that, on different real and synthetic tasks, prediction accuracy is maximized for the same hyperparameter configurations that yields the largest heterogeneity for the vertex properties of the multiplex

Summary

Introduction

Populations of spiking neurons[25]. It is well-known that several complex systems spontaneously adapt to operate towards the edge of criticality, according to a mechanism known as self-organizing criticality in the statistical physics literature[26]. Unsupervised learning is achieved by observing the dynamics of the states These methods are usually based on a statistical framework, which requires reservoir outputs to be independent and identically distributed (i.i.d.). Temporal dependencies in time series of neurons activations (states) are converted into connections of a graph representation This step allows to relax the i.i.d. assumption of statistical methods. Important properties, linking the structure of HVGs with features (e.g., onset of chaotic behavior) of the dynamic system underlying the analyzed time series, have been recently studied[42,43,44,45,46] This last aspect provides an important justification for using HVGs to model neuron activations for the purpose of hyperparameter tuning. We provide experimental evidence that our methods achieve performance comparable with supervised techniques for identifying hyperparamer configurations with high prediction accuracy and large memory capacity

Methods

Results

Conclusion