Echo-State Restricted Boltzmann Machines: A Perspective on Information Compensation

Abstract

Feature learning has been introduced in the modeling of echo state networks (ESNs). However, the procedure of feature learning is generally accompanied by certain information loss. In this regard, this paper proposes a hybrid neural network model from the information compensation view, named echo-state restricted Boltzmann machine (ERBM). It is deemed as a unified and coherent architecture with the successive functionalities of feature learning, information compensation, input superposition, and supervised nonlinear approximation. This is the first systematic model seeking to improve the ESN representative by that a direct weighted compensation channel is built to enhance feature learning. On the widely used benchmarks, we demonstrate that ERBM is fully competent to handle the nonlinear approximation tasks and superior to the state of the art in nonlinear approximation, robustness, and memory capacity.