Fading Memory Property Research Articles

An Antiferromagnetic Neuromorphic Memory Based on Perpendicularly Magnetized CoO.

Antiferromagnets (AFMs) are ideal materials to boost neuromorphic computing toward the ultrahigh speed and ultracompact integration regime. However, developing a suitable AFM neuromorphic memory remains an aspirational but challenging goal. In this work, we construct such a memory based on the CoO/Pt heterostructure, in which the collinear insulating AFM CoO shows a strong perpendicular anisotropy facilitating its electrical readout and writing. Utilizing the unique nonlinear response and bipolar fading memory properties of the device, we demonstrate a multidimensional reservoir computing beyond the traditional binary paradigm. These results are expected to pave the way toward next-generation fast and massive neuromorphic computing.

Highly-integrable analogue reservoir circuits based on a simple cycle architecture

Physical reservoir computing is a promising solution for accelerating artificial intelligence (AI) computations. Various physical systems that exhibit nonlinear and fading-memory properties have been proposed as physical reservoirs. Highly-integrable physical reservoirs, particularly for edge AI computing, has a strong demand. However, realizing a practical physical reservoir with high performance and integrability remains challenging. Herein, we present an analogue circuit reservoir with a simple cycle architecture suitable for complementary metal-oxide-semiconductor (CMOS) chip integration. In several benchmarks and demonstrations using synthetic and real-world data, our developed hardware prototype and its simulator exhibit a high prediction performance and sufficient memory capacity for practical applications, showing promise for future applications in highly integrated AI accelerators.

Switching Dynamics in Anti‐Ferroelectric Transistor for Multimodal Reservoir Computing

AbstractSpatial‐temporal time series analysis and forecasting are crucial for understanding dynamic systems and making informed decisions. Recurrent neural networks (RNNs) have paved the way for reservoir computing (RC), a method enabling effective temporal information processing at low training costs. While software‐based RC performs well, physical RC systems face challenges like slow processing speed and limited state richness, leading to high hardware costs. This study introduces an innovative approach, i.e., the antiferroelectric field effect transistor‐based RC (AFeFET‐based RC) system for efficient temporal data processing. By exploiting the fading memory property inherent in hafnium oxide‐based antiferroelectric material, this system demonstrates promise for physical RC implementation. Moreover, it leverages the light sensitivity of 2D molybdenum disulfide (MoS2) channels for controllable temporal dynamics under electrical and optical stimuli. This dual‐mode modulation significantly enriches the reservoir state, boosting overall system performance. Experimental tests on standard benchmarking tasks using the AFeFET‐based RC system yielded impressive accuracy results (95.4%) in spoken‐digit recognition and a remarkable normalized root mean square error (NRMSE) of 0.015 in Mackey–Glass time series prediction.

Fading memory as inductive bias in residual recurrent networks

Residual connections have been proposed as an architecture-based inductive bias to mitigate the problem of exploding and vanishing gradients and increased task performance in both feed-forward and recurrent networks (RNNs) when trained with the backpropagation algorithm. Yet, little is known about how residual connections in RNNs influence their dynamics and fading memory properties. Here, we introduce weakly coupled residual recurrent networks (WCRNNs) in which residual connections result in well-defined Lyapunov exponents and allow for studying properties of fading memory. We investigate how the residual connections of WCRNNs influence their performance, network dynamics, and memory properties on a set of benchmark tasks. We show that several distinct forms of residual connections yield effective inductive biases that result in increased network expressivity. In particular, those are residual connections that (i) result in network dynamics at the proximity of the edge of chaos, (ii) allow networks to capitalize on characteristic spectral properties of the data, and (iii) result in heterogeneous memory properties. In addition, we demonstrate how our results can be extended to non-linear residuals and introduce a weakly coupled residual initialization scheme that can be used for Elman RNNs.

Edge of Stability Echo State Network.

Echo state networks (ESNs) are time series processing models working under the echo state property (ESP) principle. The ESP is a notion of stability that imposes an asymptotic fading of the memory of the input. On the other hand, the resulting inherent architectural bias of ESNs may lead to an excessive loss of information, which in turn harms the performance in certain tasks with long short-term memory requirements. To bring together the fading memory property and the ability to retain as much memory as possible, in this article, we introduce a new ESN architecture called the Edge of Stability ESN (). The introduced model is based on defining the reservoir layer as a convex combination of a nonlinear reservoir (as in the standard ESN), and a linear reservoir that implements an orthogonal transformation. In virtue of a thorough mathematical analysis, we prove that the whole eigenspectrum of the Jacobian of the map can be contained in an annular neighborhood of a complex circle of controllable radius. This property is exploited to tune the 's dynamics close to the edge-of-chaos regime by design. Remarkably, our experimental analysis shows that model can reach the theoretical maximum short-term memory capacity (MC). At the same time, in comparison to conventional reservoir approaches, is shown to offer an excellent trade-off between memory and nonlinearity, as well as a significant improvement of performance in autoregressive nonlinear modeling and real-world time series modeling.

In materia reservoir computing with a fully memristive architecture based on self-organizing nanowire networks.

Neuromorphic computing aims at the realization of intelligent systems able to process information similarly to our brain. Brain-inspired computing paradigms have been implemented in crossbar arrays of memristive devices; however, this approach does not emulate the topology and the emergent behaviour of biological neuronal circuits, where the principle of self-organization regulates both structure and function. Here, we report on in materia reservoir computing in a fully memristive architecture based on self-organized nanowire networks. Thanks to the functional synaptic connectivity with nonlinear dynamics and fading memory properties, the designless nanowire complex network acts as a network-wide physical reservoir able to map spatio-temporal inputs into a feature space that can be analysed by a memristive resistive switching memory read-out layer. Computing capabilities, including recognition of spatio-temporal patterns and time-series prediction, show that the emergent memristive behaviour of nanowire networks allows in materia implementation of brain-inspired computing paradigms characterized by a reduced training cost.

Fading memory echo state networks are universal

Echo state networks (ESNs) have been recently proved to be universal approximants for input/output systems with respect to various Lp-type criteria. When 1≤p<∞, only p-integrability hypotheses need to be imposed, while in the case p=∞ a uniform boundedness hypotheses on the inputs is required. This note shows that, in the last case, a universal family of ESNs can be constructed that contains exclusively elements that have the echo state and the fading memory properties. This conclusion could not be drawn with the results and methods available so far in the literature.

Direct numerical simulations of turbulent viscoelastic jets

Direct numerical simulations (DNS) of spatially evolving turbulent planar jets of viscoelastic fluids described by the FENE-P model, such as those consisting of a Newtonian fluid solvent carrying long chain polymer molecules, are carried out in order to develop a theory for the far field of turbulent jets of viscoelastic fluids. New evolution relations for the jet shear-layer thickness , centreline velocity and maximum polymer stresses are derived and validated by the new DNS data, yielding , , and , respectively, where is the coordinate in the streamwise direction. It is shown that, compared with a classical (Newtonian) turbulent jet, the effect of the polymers is to reduce the spreading rate, centreline velocity decay, Reynolds stresses and viscous dissipation rate. The self-preserving character of the flow is analysed and it is shown that profiles of mean velocity, Reynolds stresses and polymer stresses are self-similar provided the proper scales are used in the normalisation of these quantities. A fundamental difference from the Newtonian jet in this regard is the necessity for two, instead of only one, different velocity and length scales to properly characterise the evolution of the turbulent flow. These extra velocity and length scales are directly related to a time scale associated with the characteristic fading memory property of viscoelastic fluids.

The reservoir's perspective on generalized synchronization.

We employ reservoir computing for a reconstruction task in coupled chaotic systems, across a range of dynamical relationships including generalized synchronization. For a drive-response setup, a temporal representation of the synchronized state is discussed as an alternative to the known instantaneous form. The reservoir has access to both representations through its fading memory property, each with advantages in different dynamical regimes. We also extract signatures of the maximal conditional Lyapunov exponent in the performance of variations of the reservoir topology. Moreover, the reservoir model reproduces different levels of consistency where there is no synchronization. In a bidirectional coupling setup, high reconstruction accuracy is achieved despite poor observability and independent of generalized synchronization.

Reservoir Computing Universality With Stochastic Inputs.

The universal approximation properties with respect to L p -type criteria of three important families of reservoir computers with stochastic discrete-time semi-infinite inputs are shown. First, it is proven that linear reservoir systems with either polynomial or neural network readout maps are universal. More importantly, it is proven that the same property holds for two families with linear readouts, namely, trigonometric state-affine systems and echo state networks, which are the most widely used reservoir systems in applications. The linearity in the readouts is a key feature in supervised machine learning applications. It guarantees that these systems can be used in high-dimensional situations and in the presence of large data sets. The L p criteria used in this paper allow the formulation of universality results that do not necessarily impose almost sure uniform boundedness in the inputs or the fading memory property in the filter that needs to be approximated.

Fastest time-series prediction rate achieved on a Boolean network reservoir

The utility of a network of Boolean nodes, with fading-memory property and real-valued input, has been demonstrated to predict the long-term behavior of a chaotic system.

Rapid time series prediction with a hardware-based reservoir computer.

Reservoir computing is a neural network approach for processing time-dependent signals that has seen rapid development in recent years. Physical implementations of the technique using optical reservoirs have demonstrated remarkable accuracy and processing speed at benchmark tasks. However, these approaches require an electronic output layer to maintain high performance, which limits their use in tasks such as time-series prediction, where the output is fed back into the reservoir. We present here a reservoir computing scheme that has rapid processing speed both by the reservoir and the output layer. The reservoir is realized by an autonomous, time-delay, Boolean network configured on a field-programmable gate array. We investigate the dynamical properties of the network and observe the fading memory property that is critical for successful reservoir computing. We demonstrate the utility of the technique by training a reservoir to learn the short- and long-term behavior of a chaotic system. We find accuracy comparable to state-of-the-art software approaches of a similar network size, but with a superior real-time prediction rate up to 160 MHz.

Echo state networks are universal

This paper shows that echo state networks are universal uniform approximants in the context of discrete-time fading memory filters with uniformly bounded inputs defined on negative infinite times. This result guarantees that any fading memory input/output system in discrete time can be realized as a simple finite-dimensional neural network-type state-space model with a static linear readout map. This approximation is valid for infinite time intervals. The proof of this statement is based on fundamental results, also presented in this work, about the topological nature of the fading memory property and about reservoir computing systems generated by continuous reservoir maps.

Evaluation of the computational capabilities of a memristive random network (MN3) under the context of reservoir computing

This work presents the simulation results of a novel recurrent, memristive neuromorphic architecture, the MN3 and explores its computational capabilities in the performance of a temporal pattern recognition task by considering the principles of the reservoir computing approach. A simple methodology based on the definitions of ordered and chaotic dynamical systems was used to determine the separation and fading memory properties of the architecture. The results show the potential use of this architecture as a reservoir for the on-line processing of time-varying inputs.

Model predictive control of partially fading memory systems with binary inputs

This paper presents a heuristic algorithm to implement a model predictive controller for systems with binary inputs in which the effect of the control signal on the response partially vanishes before reaching steady state, for example systems that exhibit both fast and slow stable dynamics. The proposed algorithm is based on an iterative procedure that constructs a reduced set of suboptimal solutions. The size of this set can be set accordingly to the computing capabilities and the sample time. The iterative procedure rejects possible solutions profiting from the partial fading memory property of the system and an approximation of the optimal cost-to-go function. The properties of the proposed controller are illustrated with a simulated example of a photobioreactor.

Bohr-Neugebauer property for almost automorphic partial functional differential equations

ABSTRACTWe prove that a partial functional differential equation with infinite delay has the Bohr–Neugebauer property, when the semigroup generated by the differential operator is immediately compact and when the phase space has the uniform fading memory property. To illustrate our main result, we propose an application to a reaction–diffusion equation with continuous delay.

КРИВЫЕ ПОЛЗУЧЕСТИ И РЕЛАКСАЦИИ НЕЛИНЕЙНОГО ОПРЕДЕЛЯЮЩЕГО СООТНОШЕНИЯ Ю.Н. РАБОТНОВА ДЛЯ ВЯЗКОУПРУГОПЛАСТИЧНЫХ МАТЕРИАЛОВ

General qualitative properties of the theoretic creep curves and relaxation curves produced by the Rabotnov nonlinear hereditary constitutive relation with two arbitrary material functions are studied analytically in uni-axial case. Necessary restrictions that should be imposed on the material functions to provide an adequate description of typical test relaxation, creep and recovery curves of viscoelastoplastic materials are revealed and clearly formulated. The Rabotnov constitutive relation is proved to be able to simulate the basic rheological phenomena related to creep, relaxation and recovery which are typical to a wide class of viscoelastoplastic materials. In particular, it is capable to produce creep curves with flexure point and convex down arcs, it can simulate primary, secondary and tertiary creep, creep failure, tension compression asymmetry, complete and incomplete recovery, fading memory property and so on. A number of effects are pointed out that the model can't simulate whatever material functions are taken (e.g. strain dependence of relaxation curve shape and relaxation spectrum). The capabilities of the Rabotnov nonlinear constitutive relation and its applicability scope are elucidated and compared to capabilities of the Boltzmann - Volterra linear viscoelasticity equation which is the particular case the Rabotnov's relation. Keywords: viscoelastîplasticity, nonlinear constitutive equation, theoretic creep curves, creep rate, recovery, relaxation curves, tension compression asymmetry. &nbsp

Relationships Between Two Definitions of Fading Memory for Discrete-Time Systems

Abstract In this paper, we refer to two definitions of fading memory property, which were published in the literature, for discrete-time circuits and systems. One of these definitions relates to systems working with signals (sequences) defined for both the positive and negative integers, expanding from minus infinity to plus infinity. On the other hand, the second one refers to systems processing sequences defined only for nonnegative integers, that is starting at the discrete-time point equal to zero and expanding to plus infinity. We show here that the second definition follows from the first one. That is they are not independent. Moreover, we also show that if an operator describing a system possesses a fading memory according to the second definition, then its associated operator has this property, too, but in accordance with the first definition.

A note on exponential convergence of neural networks with unbounded distributed delays

This note examines issues concerning global exponential convergence of neural networks with unbounded distributed delays. Sufficient conditions are derived by exploiting exponentially fading memory property of delay kernel functions. The method is based on comparison principle of delay differential equations and does not need the construction of any Lyapunov functionals. It is simple yet effective in deriving less conservative exponential convergence conditions and more detailed componentwise decay estimates. The results of this note and [Chu T. An exponential convergence estimate for analog neural networks with delay. Phys Lett A 2001;283:113–8] suggest a class of neural networks whose globally exponentially convergent dynamics is completely insensitive to a wide range of time delays from arbitrary bounded discrete type to certain unbounded distributed type. This is of practical interest in designing fast and reliable neural circuits. Finally, an open question is raised on the nature of delay kernels for attaining exponential convergence in an unbounded distributed delayed neural network.

From Data to Nonlinear Predictive Control. 1. Identification of Multivariable Nonlinear State Observers

This work aims at the identification of a special class nonlinear state space observers for nonlinear multivariable systems directly from input−output data when the data is corrupted with unmeasured disturbances. At the identification stage, the one step ahead predictor form of the model is arranged to have a Weiner-like structure. The linear dynamic component of the predictor is parametrized using generalized orthonormal basis functions. The resulting observer is shown to be a nonlinear ARX (NARX) type model with an infinite but fading memory property. It is also shown that the proposed model structure is capable of capturing input as well as output multiplicity (multiple steady states) behavior. The efficacy of the proposed modeling scheme is demonstrated using simulation studies on a continuously stirred tank reactor (CSTR) process model, which exhibits input multiplicity, and another CSTR process model that exhibits output multiplicities. The types of unmeasured disturbances investigated are (a) unkno...