Dynamic Recurrent Neural Network Research Articles

Recurrence resonance - noise-enhanced dynamics in recurrent neural networks

Understanding how neural networks process information is a fundamental challenge in neuroscience and artificial intelligence. A pivotal question in this context is how external stimuli, particularly noise, influence the dynamics and information flow within these networks. Traditionally, noise is perceived as a hindrance to information processing, introducing randomness and diminishing the fidelity of neural signals. However, distinguishing noise from structured input uncovers a paradoxical insight: under specific conditions, noise can actually enhance information processing. This intriguing possibility prompts a deeper investigation into the nuanced role of noise within neural networks. In specific motifs of three recurrently connected neurons with probabilistic response, the spontaneous information flux, defined as the mutual information between subsequent states, has been shown to increase by adding ongoing white noise of some optimal strength to each of the neurons. However, the precise conditions for and mechanisms of this phenomenon called ‘recurrence resonance’ (RR) remain largely unexplored. Using Boltzmann machines of different sizes and with various types of weight matrices, we show that RR can generally occur when a system has multiple dynamical attractors, but is trapped in one or a few of them. In probabilistic networks, the phenomenon is bound to a suitable observation time scale, as the system could autonomously access its entire attractor landscape even without the help of external noise, given enough time. Yet, even in large systems, where time scales for observing RR in the full network become too long, the resonance can still be detected in small subsets of neurons. Finally, we show that short noise pulses can be used to transfer recurrent neural networks, both probabilistic and deterministic, between their dynamical attractors. Our results are relevant to the fields of reservoir computing and neuroscience, where controlled noise may turn out a key factor for efficient information processing leading to more robust and adaptable systems.

Assessment of Seasonal Rainfall Prediction in Ethiopia: Evaluating a Dynamic Recurrent Neural Network to Downscale ECMWF-SEAS5 Rainfall

Assessment of Seasonal Rainfall Prediction in Ethiopia: Evaluating a Dynamic Recurrent Neural Network to Downscale ECMWF-SEAS5 Rainfall

Tai Chi Practice Buffers Aging Effects in Functional Brain Connectivity.

Tai Chi (TC) practice has been shown to improve both cognitive and physical function in older adults. However, the neural mechanisms underlying the benefits of TC remain unclear. Our primary aims are to explore whether distinct age-related and TC-practice-related relationships can be identified with respect to either temporal or spatial (within/between-network connectivity) differences. This cross-sectional study examined recurrent neural network dynamics, employing an adaptive, data-driven thresholding approach to source-localized resting-state EEG data in order to identify meaningful connections across time-varying graphs, using both temporal and spatial features derived from a hidden Markov model (HMM). Mann-Whitney U tests assessed between-group differences in temporal and spatial features by age and TC practice using either healthy younger adult controls (YACs, n = 15), healthy older adult controls (OACs, n = 15), or Tai Chi older adult practitioners (TCOAs, n = 15). Our results showed that aging is associated with decreased within-network and between-network functional connectivity (FC) across most brain networks. Conversely, TC practice appears to mitigate these age-related declines, showing increased FC within and between networks in older adults who practice TC compared to non-practicing older adults. These findings suggest that TC practice may abate age-related declines in neural network efficiency and stability, highlighting its potential as a non-pharmacological intervention for promoting healthy brain aging. This study furthers the triple-network model, showing that a balancing and reorientation of attention might be engaged not only through higher-order and top-down mechanisms (i.e., FPN/DAN) but also via the coupling of bottom-up, sensory-motor (i.e., SMN/VIN) networks.

Time-optimal constrained kinematic control of robotic manipulators by recurrent neural network

Time-optimal kinematic control is a vital concern for industrial manipulators to save allocated motion task time as much as possible. This requires maximizing the end-effector velocity to minimize the time required for path tracking. Nonetheless, it remains a challenge to ensure that joint motion constraints are not violated during this process, even with the aim of maximizing end-effector velocity simultaneously. This paper introduces a novel approach, which for the first time leverages dynamic recurrent neural networks (RNNs) within a constrained optimization framework to attain time-optimal kinematic control for manipulators. The theoretical analysis of the RNN-based kinematic control solver is addressed, ensuring both its optimality and convergence for achieving time-optimal kinematic control. The proposed method enables the maximization of end-effector velocity to achieve time-optimal kinematic control without violating all joint velocity limits simultaneously. In contrast to previous kinematic control schemes, the proposed method can enhance the end-effector path tracking speed of completion by 100% around, we substantiate the effectiveness and superiority of the proposed approach via simulation and V-Rep experiment on the manipulators.

Fracture mechanism of Ti60 alloy at high temperature in very high cycle fatigue regime and fatigue life prediction

AbstractThis study aimed to investigate the failure mechanism of Ti60 titanium alloy in the high cycle fatigue (HCF) and very high cycle fatigue (VHCF) at 500°C. Ti60 specimens were characterized before and after the 500°C VHCF test, and the fracture surfaces were observed. The results show that there are three different fatigue failure mechanisms at 500°C: (i) equiaxed primary α phase (αp) cleavage, forming small unsmooth facets and rough fracture area (RA), and fusion of non‐adjacent facets to grow into the main crack. (ii) αp grains gather into large grain with triple junction, and large grain slip to form large fusion facet. (iii) Oxide shedding and then cracks form. A dynamic recurrent neural network model is used to predict the fatigue life of Ti60 alloy, 91% of the overall predictions were within the scatter band of 3.0, and 100% were within the scatter band of 5.0.

Data-Driven Self-Triggered Control for Networked Motor Control Systems Using RNNs and Pre-Training: A Hierarchical Reinforcement Learning Framework.

This paper introduces a novel data-driven self-triggered control approach based on a hierarchical reinforcement learning framework in networked motor control systems. This approach divides the self-triggered control policy into higher and lower layers, with the higher-level policy guiding the lower-level policy in decision-making, thereby reducing the exploration space of the lower-level policy and improving the efficiency of the learning process. The data-driven framework integrates with the dual-actor critic algorithm, using two interconnected neural networks to approximate the hierarchical policies. In this framework, we use recurrent neural networks as the network architecture for the critic, utilizing the temporal dynamics of recurrent neural networks to better capture the dependencies between costs, thus enhancing the critic network's efficiency and accuracy in approximating the multi-time cumulative cost function. Additionally, we have developed a pre-training method for the control policy networks to further improve learning efficiency. The effectiveness of our proposed method is validated through a series of numerical simulations.

Suppression of chaos in a partially driven recurrent neural network

The dynamics of recurrent neural networks (RNNs), and particularly their response to inputs, play a critical role in information processing. In many applications of RNNs, only a specific subset of the neurons generally receive inputs. However, it remains to be theoretically clarified how the restriction of the input to a specific subset of neurons affects the network dynamics. Considering RNNs with such restricted input, we investigate how the proportion, p, of the neurons receiving inputs (the “input neurons”) and the strength of the input signals affect the dynamics by analytically deriving the conditional maximum Lyapunov exponent. Our results show that for sufficiently large p, the maximum Lyapunov exponent decreases monotonically as a function of the input strength, indicating the suppression of chaos, but if p is smaller than a critical threshold, pc, even significantly amplified inputs cannot suppress spontaneous chaotic dynamics. Furthermore, although the value of pc is seemingly dependent on several model parameters, such as the sparseness and strength of recurrent connections, it is proved to be intrinsically determined solely by the strength of chaos in spontaneous activity of the RNN. This is to say, despite changes in these model parameters, it is possible to represent the value of pc as a common invariant function by appropriately scaling these parameters to yield the same strength of spontaneous chaos. Our study suggests that if p is above pc, we can bring the neural network to the edge of chaos, thereby maximizing its information processing capacity, by amplifying inputs. Published by the American Physical Society 2024

Application of a novel dynamic recurrent fuzzy neural network with rule self-adaptation based on chaotic quantum pigeon-inspired optimization in modeling for gas turbine

With the rapid development of renewable energy, higher requirements are put forward for the control ability of thermal power units. And an accurate system model is the premise and guarantee of controller design. In this article, a novel dynamic recurrent fuzzy neural network with rule self-adaptation modeling approach based on chaotic quantum pigeon-inspired optimization (NDRFNN-CQPIO) is proposed for the modeling of gas turbine system. The introduction of recursive layer, automatic generation of fuzzy rules and fast parameter solving are the main features of the proposed algorithm. First, set the appropriate threshold, and process the system error and distance index through 4 preset cases to generate fuzzy rules. When a new rule is generated, its antecedent parameters are determined at the same time, and define a boundary to determine whether the antecedent parameters need to be adjusted, so as to avoid the generation of redundant Gaussian functions. Then, a pruning technique based on the contributions of the normalized neuron and learning error is raised to trim the redundant fuzzy rules to ensure the model structure more compact and efficient. Finally, using the chaotic quantum pigeon-inspired algorithm to solve the consequent parameters quickly and accurately. The actual operation data of a gas turbine system were selected to identify and verify the model. In the established model, the root mean square error of output power, exhaust gas temperature, rotor speed, and exhaust gas flow are 0.0023, 0.0015, 0.0011 and 0.0069, respectively. The simulation results show that the proposed approach is effective and feasible, and it can be used for the model identification of gas turbine system. Moreover, we conducted robustness analysis of the model and proved that the proposed model has strong robustness, which provides a basis for its further application.

Development of Mathematical Models for Industrial Processes Using Dynamic Neural Networks

Dynamic neural networks (DNNs) are a type of artificial neural network (ANN) designed to work with sequential data where context in time is important. Unlike traditional static neural networks that process data in a fixed order, dynamic neural networks use information about past inputs, which is important if the dynamic of a certain process is emphasized. They are commonly used in natural language processing, speech recognition, and time series prediction. In industrial processes, their use is interesting for the prediction of difficult-to-measure process variables. In an industrial isomerization process, it is crucial to measure the quality attributes that affect the octane number of gasoline. Process analyzers commonly used for this purpose are expensive and subject to failure. Therefore, to achieve continuous production in the event of a malfunction, mathematical models for estimating product quality attributes are imposed as a solution. In this paper, mathematical models were developed using dynamic recurrent neural networks (RNNs), i.e., their subtype of a long short-term memory (LSTM) architecture. The results of the developed models were compared with the results of several types of other data-driven models developed for an isomerization process, such as multilayer perceptron (MLP) artificial neural networks, support vector machines (SVM), and dynamic polynomial models. The obtained results are satisfactory, suggesting a good possibility of application.

Convolutional gated recurrent unit-driven multidimensional dynamic graph neural network for subject-independent emotion recognition

Electroencephalogram (EEG) could directly reflect human brain activities. Recently, EEG-based emotion recognition technology has attracted widespread attention. However, although most of the existing methods can extract temporal and spatial features of EEG signals, the extraction process of the time-series information and spatial features is separate from each other. Some spatial features may be lost when extracting time-series information, and vice versa. Additionally, the non-Euclidean representations between the EEG electrodes cannot be ignored. In response to these issues, we construct a convolutional gated recurrent unit-driven multidimensional dynamic graph neural network (CGRU-MDGN), which mines its local spatial characteristics of EEG signals while learning the temporal information and captures non-Euclidean spatial features between EEG channels to classify emotions more accurately. Specifically, we substitute the fully connected layers of the GRU with convolution layers, which can take into account both the local spatial features and time-series information of EEG signals. Next, multiple non-Euclidean spatial representations are generated using multidimensional dynamic graph convolution. Furthermore, the characteristics of different dimensions or views are merged through the adaptive-weighted summation technique to realize the maximized feature extraction as much as possible. Comprehensive experiments reveal that the constructed CGRU-MDGN could achieve competitive recognition results on subject-independent emotion recognition tasks.

Internal dynamics of recurrent neural networks trained to generate complex spatiotemporal patterns.

How complex patterns generated by neural systems are represented in individual neuronal activity is an essential problem in computational neuroscience as well as machine learning communities. Here, based on recurrent neural networks in the form of feedback reservoir computers, we show microscopic features resulting in generating spatiotemporal patterns including multicluster and chimera states. We show the effect of individual neural trajectories as well as whole-network activity distributions on exhibiting particular regimes. In addition, we address the question how trained output weights contribute to the autonomous multidimensional dynamics.

Guest Editorial: Special issue on recurrent dynamic neural networks: Theory and applications

Guest Editorial: Special issue on recurrent dynamic neural networks: Theory and applications

Neural Network Model-Based Control for Manipulator: An Autoencoder Perspective.

Recently, neural network model-based control has received wide interests in kinematics control of manipulators. To enhance learning ability of neural network models, the autoencoder method is used as a powerful tool to achieve deep learning and has gained success in recent years. However, the performance of existing autoencoder approaches for manipulator control may be still largely dependent on the quality of data, and for extreme cases with noisy data it may even fail. How to incorporate the model knowledge into the autoencoder controller design with an aim to increase the robustness and reliability remains a challenging problem. In this work, a sparse autoencoder controller for kinematic control of manipulators with weights obtained directly from the robot model rather than training data is proposed for the first time. By encoding and decoding the control target though a new dynamic recurrent neural network architecture, the control input can be solved through a new sparse optimization formulation. In this work, input saturation, which holds for almost all practical systems but usually is ignored for analysis simplicity, is also considered in the controller construction. Theoretical analysis and extensive simulations demonstrate that the proposed sparse autoencoder controller with input saturation can make the end-effector of the manipulator system track the desired path efficiently. Further performance comparison and evaluation against the additive noise and parameter uncertainty substantiate robustness of the proposed sparse autoencoder manipulator controller.

Multiplexing working memory and time in the trajectories of neural networks.

Working memory (WM) and timing are generally considered distinct cognitive functions, but similar neural signatures have been implicated in both. To explore the hypothesis that WM and timing may rely on shared neural mechanisms, we used psychophysical tasks that contained either task-irrelevant timing or WM components. In both cases, the task-irrelevant component influenced performance. We then developed recurrent neural network (RNN) simulations that revealed that cue-specific neural sequences, which multiplexed WM and time, emerged as the dominant regime that captured the behavioural findings. During training, RNN dynamics transitioned from low-dimensional ramps to high-dimensional neural sequences, and depending on task requirements, steady-state or ramping activity was also observed. Analysis of RNN structure revealed that neural sequences relied primarily on inhibitory connections, and could survive the deletion of all excitatory-to-excitatory connections. Our results indicate that in some instances WM is encoded in time-varying neural activity because of the importance of predicting when WM will be used.

DRRNets: Dynamic Recurrent Routing via Low-Rank Regularization in Recurrent Neural Networks.

Recurrent neural networks (RNNs) continue to show outstanding performance in sequence learning tasks such as language modeling, but it remains difficult to train RNNs for long sequences. The main challenges lie in the complex dependencies, gradient vanishing or exploding, and low resource requirement in model deployment. In order to address these challenges, we propose dynamic recurrent routing neural networks (DRRNets), which can: 1) shorten the recurrent lengths by allocating recurrent routes dynamically for different dependencies and 2) reduce the number of parameters significantly by imposing low-rank constraints on the fully connected layers. A novel optimization algorithm via low-rank constraint and sparsity projection is developed to train the network. We verify the effectiveness of the proposed method by comparing it with multiple competitive approaches in several popular sequential learning tasks, such as language modeling and speaker recognition. The results in terms of different criteria demonstrate the superiority of our proposed method.

Graph Rules for Recurrent Neural Network Dynamics

Graph Rules for Recurrent Neural Network Dynamics

Near‐optimal control of a class of output‐constrained systems using recurrent neural network: A control‐barrier function approach

AbstractThis paper proposes a near‐optimal controller design for the constrained nonlinear affine systems based on a Recurrent Neural Network (RNN) and Extended State Observers (ESOs). For this purpose, after defining a finite‐horizon integral‐type performance index, the prediction over the horizon is performed using the Taylor expansion that converts the primary problem into a finite‐dimensional optimization. In comparison with other controllers of the similar structure, the proposed method is capable of dealing with output constraints by employing the Control Barrier Function (CBF). The class of the output and input constraints are of the box‐type. Moreover, whereas several safe control approaches are proposed regardless of the performance of the closed‐loop system, this paper aims at achieving a near‐optimal performance as far as the constraints permit. As a result, a constrained optimization problem is achieved, where the online solution is obtained using a rapidly convergent RNN. Stability and the ease of implementation are some of the advantages of this network making the algorithm more reliable. Moreover, integrated stability analysis of the closed‐loop system that includes the dynamic RNN reveals that the closed‐loop system is stable in the sense of the Lyapunov stability theory. The effectiveness of the proposed control method in terms of the tracking performance and constraint satisfaction is illustrated through a simulating example of two‐inverted pendulums system. The results indicated advantages of the proposed method as compared with the recently published methods in well‐known literature.

A Dynamic Recurrent Neural Network for Predicting Higher Heating Value of Biomass.

The higher heating value (HHV) is the main property showing the energy amount of biomass samples. Several linear correlations based on either the proximate or the ultimate analysis have already been proposed for predicting biomass HHV. Since the HHV relationship with the proximate and ultimate analyses is not linear, nonlinear models might be a better alternative. Accordingly, this study employed the Elman recurrent neural network (ENN) to anticipate the HHV of different biomass samples from both the ultimate and proximate compositional analyses as the model inputs. The number of hidden neurons and the training algorithm were determined in such a way that the ENN model showed the highest prediction and generalization accuracy. The single hidden layer ENN with only four nodes, trained by the Levenberg-Marquardt algorithm, was identified as the most accurate model. The proposed ENN exhibited reliable prediction and generalization performance for estimating 532 experimental HHVs with a low mean absolute error of 0.67 and a mean square error of 0.96. In addition, the proposed ENN model provides a ground to clearly understand the dependency of the HHV on the fixed carbon, volatile matter, ash, carbon, hydrogen, nitrogen, oxygen, and sulfur content of biomass feedstocks.

Neural population dynamics of computing with synaptic modulations.

In addition to long-timescale rewiring, synapses in the brain are subject to significant modulation that occurs at faster timescales that endow the brain with additional means of processing information. Despite this, models of the brain like recurrent neural networks (RNNs) often have their weights frozen after training, relying on an internal state stored in neuronal activity to hold task-relevant information. In this work, we study the computational potential and resulting dynamics of a network that relies solely on synapse modulation during inference to process task-relevant information, the multi-plasticity network (MPN). Since the MPN has no recurrent connections, this allows us to study the computational capabilities and dynamical behavior contributed by synapses modulations alone. The generality of the MPN allows for our results to apply to synaptic modulation mechanisms ranging from short-term synaptic plasticity (STSP) to slower modulations such as spike-time dependent plasticity (STDP). We thoroughly examine the neural population dynamics of the MPN trained on integration-based tasks and compare it to known RNN dynamics, finding the two to have fundamentally different attractor structure. We find said differences in dynamics allow the MPN to outperform its RNN counterparts on several neuroscience-relevant tests. Training the MPN across a battery of neuroscience tasks, we find its computational capabilities in such settings is comparable to networks that compute with recurrent connections. Altogether, we believe this work demonstrates the computational possibilities of computing with synaptic modulations and highlights important motifs of these computations so that they can be identified in brain-like systems.

Spatio-temporal prediction of temperature in fluidized bed biomass gasifier using dynamic recurrent neural network method

• Long-short term memory model is proposed to predict temperatures in fluidized bed gasifier. • The proposed model is validated with pilot-scale plant data resolved in space and time. • Multi-step ahead temperature progressions of freeboard, fluidized bed and outlet gas, are predicted. • Gated recurrent unit and long short-term memory models are compared. In this study, a long short-term memory (LSTM) based dynamic recurrent neural network model is proposed for multi-step ahead temperature predictions in a pilot-scale fluidized bed biomass gasifier (FBG). The LSTM model predicts not only the temporal but also the spatial distribution of temperature by considering the temperature of each region of the FBG (fluidized bed, freeboard and outlet gas) as a separate target parameter. The proposed model is validated by comparing simulation data with experimental observations acquired during operation of the FBG. The validation results reveal that the proposed LSTM model is capable of accurately (MAE<6) predicting 1-min-ahead temperature of all the FBG regions. The LSTM model is further challenged for temperature predictions at farther future points (3 min and 5 min ahead) to test the prediction limits of the LSTM model. For 5 min ahead predictions, the proposed LSTM-based prediction model is also compared with other state-of-the-art dynamic neural network methods that include the standard recurrent neural network (S-RNN) and its advanced variant, the gated recurrent unit (GRU). The comparative findings for far future predictions show that LSTM has the highest accuracy, and also exhibit that GRU does not have universally faster convergence than LSTM.