Abstract
Reservoir computing is a brain-inspired machine learning framework that employs a signal-driven dynamical system, in particular harnessing common-signal-induced synchronization which is a widely observed nonlinear phenomenon. Basic understanding of a working principle in reservoir computing can be expected to shed light on how information is stored and processed in nonlinear dynamical systems, potentially leading to progress in a broad range of nonlinear sciences. As a first step toward this goal, from the viewpoint of nonlinear physics and information theory, we study the memory-nonlinearity trade-off uncovered by Dambre et al. (2012). Focusing on a variational equation, we clarify a dynamical mechanism behind the trade-off, which illustrates why nonlinear dynamics degrades memory stored in dynamical system in general. Moreover, based on the trade-off, we propose a mixture reservoir endowed with both linear and nonlinear dynamics and show that it improves the performance of information processing. Interestingly, for some tasks, significant improvements are observed by adding a few linear dynamics to the nonlinear dynamical system. By employing the echo state network model, the effect of the mixture reservoir is numerically verified for a simple function approximation task and for more complex tasks.
Highlights
A variety of dynamical systems, including recurrent neural networks, soft material, and optoelectronic and quantum systems, exhibit common-signal-induced synchronization[1,2,3,4]
Little is known about its working principle, and there are few theoretical answers to the following fundamental question: what characteristics of a dynamical system are crucial for high-performance information processing? Progress in theoretical research on reservoir computing (RC) could uncover a reservoir design principle, and deepen our understanding of information processing in dynamical systems, in particular give an answer to the question, such as how dynamical systems store and process information, discussed in a community of nonlinear physics[19]
What sort of dynamical system is preferable for the reservoir that realizes the universal transformation of the input signal and possess the appropriate memory capacity? The pioneering works in this direction tackled to find the answer; Butcher et al.[29,30,31] introduced RC with random static projection (R2SP) and Extreme Learning Machines with a time delay based on the discussion on the trade-off, and reported these architectures improves performance well for some tasks compared with the standard echo state network model
Summary
Reservoir computing is a brain-inspired machine learning framework that employs a signal-driven dynamical system, in particular harnessing common-signal-induced synchronization which is a widely observed nonlinear phenomenon. It can be expected that there exists some trade-off relation between linearity and nonlinearity in reservoir dynamics, which is required respectively for memory capacity and for the general information processing. The pioneering works in this direction tackled to find the answer; Butcher et al.[29,30,31] introduced RC with random static projection (R2SP) and Extreme Learning Machines with a time delay based on the discussion on the trade-off, and reported these architectures improves performance well for some tasks compared with the standard echo state network model. Vinckier et al.[15] introduced a linear optical dynamics on a photonic chip with nonlinear readout and showed that it possesses a remarkably high (total) memory capacity, and interestingly, exhibits high-performances for the complex tasks. We verify the effect of the mixture reservoir for more practical and complex tasks: time series forecasting of the Santa Fe Laser data set[32] and the NARMA task
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
