Training Recurrent Neural Networks Research Articles

Topological features of spike trains in recurrent spiking neural networks that are trained to generate spatiotemporal patterns.

In this study, we focus on training recurrent spiking neural networks to generate spatiotemporal patterns in the form of closed two-dimensional trajectories. Spike trains in the trained networks are examined in terms of their dissimilarity using the Victor-Purpura distance. We apply algebraic topology methods to the matrices obtained by rank-ordering the entries of the distance matrices, specifically calculating the persistence barcodes and Betti curves. By comparing the features of different types of output patterns, we uncover the complex relations between low-dimensional target signals and the underlying multidimensional spike trains.

Leakage monitoring and diagnosis of LNG storage tanks with temperature sensing network integration and artificial intelligence algorithm

Temperature is the most important safety monitoring indicator for leakage diagnosis during the operation phase of liquefied natural gas (LNG) storage tanks. Timely monitoring and accurate identification of LNG leakage events are crucial for accident prevention, loss reduction, and facility safety maintenance. This study integrates artificial intelligence (AI) algorithms and temperature sensing data to achieve intelligent monitoring and diagnosis of leakage in LNG storage tanks. Firstly, a comprehensive temperature sensing network is constructed by combining numerical simulation of the temperature field and temperature sensing experiments using fiber Bragg grating sensors. Secondly, Python is used to perform linear grid interpolation and flattening on the sensing network, generating 2D temperature nephogram samples that are conducive to AI algorithm recognition. Finally, sample features are extracted using machine vision, and leakage location calculation, leakage diagnosis and leakage volume calculation are implemented with the help of machine learning algorithms, achieving satisfactory accuracy on the test set. In addition, the ConvLSTM framework is introduced for deep learning and recurrent neural network training, enabling spatiotemporal prediction of the leakage area.

Trained recurrent neural networks develop phase-locked limit cycles in a working memory task.

Neural oscillations are ubiquitously observed in many brain areas. One proposed functional role of these oscillations is that they serve as an internal clock, or 'frame of reference'. Information can be encoded by the timing of neural activity relative to the phase of such oscillations. In line with this hypothesis, there have been multiple empirical observations of such phase codes in the brain. Here we ask: What kind of neural dynamics support phase coding of information with neural oscillations? We tackled this question by analyzing recurrent neural networks (RNNs) that were trained on a working memory task. The networks were given access to an external reference oscillation and tasked to produce an oscillation, such that the phase difference between the reference and output oscillation maintains the identity of transient stimuli. We found that networks converged to stable oscillatory dynamics. Reverse engineering these networks revealed that each phase-coded memory corresponds to a separate limit cycle attractor. We characterized how the stability of the attractor dynamics depends on both reference oscillation amplitude and frequency, properties that can be experimentally observed. To understand the connectivity structures that underlie these dynamics, we showed that trained networks can be described as two phase-coupled oscillators. Using this insight, we condensed our trained networks to a reduced model consisting of two functional modules: One that generates an oscillation and one that implements a coupling function between the internal oscillation and external reference. In summary, by reverse engineering the dynamics and connectivity of trained RNNs, we propose a mechanism by which neural networks can harness reference oscillations for working memory. Specifically, we propose that a phase-coding network generates autonomous oscillations which it couples to an external reference oscillation in a multi-stable fashion.

Brain control of bimanual movement enabled by recurrent neural networks

Brain-computer interfaces have so far focused largely on enabling the control of a single effector, for example a single computer cursor or robotic arm. Restoring multi-effector motion could unlock greater functionality for people with paralysis (e.g., bimanual movement). However, it may prove challenging to decode the simultaneous motion of multiple effectors, as we recently found that a compositional neural code links movements across all limbs and that neural tuning changes nonlinearly during dual-effector motion. Here, we demonstrate the feasibility of high-quality bimanual control of two cursors via neural network (NN) decoders. Through simulations, we show that NNs leverage a neural ‘laterality’ dimension to distinguish between left and right-hand movements as neural tuning to both hands become increasingly correlated. In training recurrent neural networks (RNNs) for two-cursor control, we developed a method that alters the temporal structure of the training data by dilating/compressing it in time and re-ordering it, which we show helps RNNs successfully generalize to the online setting. With this method, we demonstrate that a person with paralysis can control two computer cursors simultaneously. Our results suggest that neural network decoders may be advantageous for multi-effector decoding, provided they are designed to transfer to the online setting.

SFR Modeling for Hybrid Power Systems Based on Deep Transfer Learning

A deep transfer learning method is presented for establishing the aggregated system frequency response (SFR) model of wind-thermal hybrid power systems (HPSs). In order to deal with nonlinearities and non-Gaussian disturbances, the quadratic survival information potential (QSIP) of the squared identification error is employed to construct the performance index when training recurrent neural networks (RNNs). A pre-trained SFR model is then obtained by the improved RNNs using the source domain data collected from the HPS in historical scenarios. Subsequently, the maximum mean difference is utilized to test the similarity of the HPS in historical and current scenarios. After that, the pre-trained SFR model is fine-tuned by adding some nodes to the recurrent layer and a functional link to the input layer. The SFR model of the HPS operating in current scenario can then be built based on the transferred source domain pre-trained SFR model. Simulation results illustrate that the proposed data driven modelling method can obtain accurate, effective and timely SFR model for a wind-thermal HPS with different wind speeds and load disturbances.

Enhancing bearing fault diagnosis using motor current signals: A novel approach combining time shifting and CausalConvNets

In motor drive system, Bearing fault detection through motor current signal (MCS) analysis has gained recognition for its cost-effectiveness and non-invasive nature. However, two-dimensional (2D) convolutional neural networks (CNNs) exhibit suboptimal performance when directly applied to raw stator current signals for fault detection. Meanwhile, the training of Recurrent Neural Networks (RNNs) for processing one-dimensional (1D) temporal sequence can be a time-consuming task due to the inherent procedures of the network structure. To balance time-consuming and accuracy, this paper presents a novel MCS-based bearing fault detection model utilizing a 1D CNN and corresponding pre-processing methods. In the early stages, a noise cancellation strategy termed time shifting is implemented. Subsequently, wavelet transform is employed to further refine the processed signal by eliminating components with minimal significance. By compressing homogenous signals, the resultant tensor, referred to as the “root map”, is employed as the input for the proposed temporal causal network. Two groups of experimental results conducted on both laboratory and industrial conditions validate the effectiveness and convenience of the proposed methodology.

Toward Effective Aircraft Call Sign Detection Using Fuzzy String-Matching between ASR and ADS-B Data

Recently, artificial intelligence and data science have witnessed dramatic progress and rapid growth, especially Automatic Speech Recognition (ASR) technology based on Hidden Markov Models (HMMs) and Deep Neural Networks (DNNs). Consequently, new end-to-end Recurrent Neural Network (RNN) toolkits were developed with higher speed and accuracy that can often achieve a Word Error Rate (WER) below 10%. These toolkits can nowadays be deployed, for instance, within aircraft cockpits and Air Traffic Control (ATC) systems in order to identify aircraft and display recognized voice messages related to flight data, especially for airports not equipped with radar. Hence, the performance of air traffic controllers and pilots can ultimately be improved by reducing workload and stress and enforcing safety standards. Our experiment conducted at Tangier’s International Airport ATC aimed to build an ASR model that is able to recognize aircraft call signs in a fast and accurate way. The acoustic and linguistic models were trained on the Ibn Battouta Speech Corpus (IBSC), resulting in an unprecedented speech dataset with approved transcription that includes real weather aerodrome observation data and flight information with a call sign captured by an ADS-B receiver. All of these data were synchronized with voice recordings in a structured format. We calculated the WER to evaluate the model’s accuracy and compared different methods of dataset training for model building and adaptation. Despite the high interference in the VHF radio communication channel and fast-speaking conditions that increased the WER level to 20%, our standalone and low-cost ASR system with a trained RNN model, supported by the Deep Speech toolkit, was able to achieve call sign detection rate scores up to 96% in air traffic controller messages and 90% in pilot messages while displaying related flight information from ADS-B data using the Fuzzy string-matching algorithm.

Statistical physics of learning in high-dimensional chaotic systems

In many complex systems, elementary units live in a chaotic environment and need to adapt their strategies to perform a task by extracting information from the environment and controlling the feedback loop on it. One of the main examples of systems of this kind is provided by recurrent neural networks. In this case, recurrent connections between neurons drive chaotic behavior, and when learning takes place, the response of the system to a perturbation should also take into account its feedback on the dynamics of the network itself. In this work, we consider an abstract model of a high-dimensional chaotic system as a paradigmatic model and study its dynamics. We study the model under two particular settings: Hebbian driving and FORCE training. In the first case, we show that Hebbian driving can be used to tune the level of chaos in the dynamics, and this reproduces some results recently obtained in the study of more biologically realistic models of recurrent neural networks. In the latter case, we show that the dynamical system can be trained to reproduce simple periodic functions. To do this, we consider the FORCE algorithm—originally developed to train recurrent neural networks—and adapt it to our high-dimensional chaotic system. We show that this algorithm drives the dynamics close to an asymptotic attractor the larger the training time. All our results are valid in the thermodynamic limit due to an exact analysis of the dynamics through dynamical mean field theory.

Research of Carbon Emission Prediction: An Oscillatory Particle Swarm Optimization for Long Short-Term Memory

Carbon emissions play a significant role in shaping social policy-making, industrial planning, and other critical areas. Recurrent neural networks (RNNs) serve as the major choice for carbon emission prediction. However, year-frequency carbon emission data always results in overfitting during RNN training. To address this issue, we propose a novel model that combines oscillatory particle swarm optimization (OPSO) with long short-term memory (LSTM). OPSO is employed to fine-tune the hyperparameters of LSTM, utilizing an oscillatory strategy to effectively mitigate overfitting and consequently improve the accuracy of the LSTM model. In validation tests, real data from Hainan Province, encompassing diverse dimensions such as gross domestic product, forest area, and ten other relevant factors, are used. Standard LSTM and PSO-LSTM are selected in the control group. The mean absolute error (MAE), root mean square error (RMSE), and mean absolute percentage error (MAPE) are used to evaluate the performance of these methods. In the test dataset, the MAE of OPSO-LSTM is 117.708, 65.72% better than LSTM and 29.48% better than PSO-LSTM. The RMSE of OPSO-LSTM is 149.939, 68.52% better than LSTM and 41.90% better than PSO-LSTM. The MAPE of OPSO-LSTM is 0.017, 65.31% better than LSTM, 29.17% better than PSO-LSTM. The experimental results prove that OPSO-LSTM can provide reliable predictions for carbon emissions.

Privacy-Preserving Federated Recurrent Neural Networks

We present RHODE, a novel system that enables privacy-preserving training of and prediction on Recurrent Neural Networks (RNNs) in a cross-silo federated learning setting by relying on multiparty homomorphic encryption. RHODE preserves the confidentiality of the training data, the model, and the prediction data; and it mitigates federated learning attacks that target the gradients under a passive-adversary threat model. We propose a packing scheme, multi-dimensional packing, for a better utilization of Single Instruction, Multiple Data (SIMD) operations under encryption. With multi-dimensional packing, RHODE enables the efficient processing, in parallel, of a batch of samples. To avoid the exploding gradients problem, RHODE provides several clipping approximations for performing gradient clipping under encryption. We experimentally show that the model performance with RHODE remains similar to non-secure solutions both for homogeneous and heterogeneous data distributions among the data holders. Our experimental evaluation shows that RHODE scales linearly with the number of data holders and the number of timesteps, sub-linearly and sub-quadratically with the number of features and the number of hidden units of RNNs, respectively. To the best of our knowledge, RHODE is the first system that provides the building blocks for the training of RNNs and its variants, under encryption in a federated learning setting.

A High-Security Probabilistic Constellation Shaping Transmission Scheme Based on Recurrent Neural Networks

In this paper, a high-security probabilistic constellation shaping transmission scheme based on recurrent neural networks (RNNs) is proposed, in which the constellation point probabilistic distribution is generated based on recurrent neural network training. A 4D biplane fractional-order chaotic system is introduced to ensure the security performance of the system. The performance of the proposed scheme is verified in a 2 km seven-core optical transmission system. The RNN-trained probabilistic shaping scheme achieves a transmission gain of 1.23 dB compared to the standard 16QAM signal, 0.39 dB compared to the standard Maxwell-Boltzmann (M-B) distribution signal, and a higher net bit rate. The proposed encryption scheme has higher randomness and security than the conventional integer-order chaotic system, with a key space of 10,163. This scheme will have a promising future fiber optic transmission scheme because it combines the efficient transmission and security of fiber optic transmission systems.

Continuous self‐adversarial training of recurrent neural network–based constitutive description

AbstractData‐driven methods yield advantages in computational homogenization approaches due to the ability to capture complex material behaviour without the necessity to assume specific constitutive models. Neural network–based constitutive descriptions are one of the most widely used data‐driven approaches in the context of computational mechanics. The accuracy of this method strongly depends on the available data. Additionally, when considering inelastic materials, whose constitutive responses depend on the loading history, the accuracy and robustness of the approximation are influenced by the training algorithm. The applied recurrent neural networks exhibit reduced robustness in the presence of errors in the input. When capturing the history dependency using previously predicted material responses, occurrences of prediction errors accumulate over several time steps. An approach for achieving enhanced robustness of the predictions is based on extending the initial training dataset by iteratively generating adversarial examples, subjected to perturbations, based on the current prediction errors. In this contribution, a continuous self‐adversarial training approach yielding robust recurrent neural network constitutive descriptions for inelastic materials is presented. Compared to the iterative method it is based on, it exhibits significantly improved training efficiency. In order to demonstrate the capabilities of the proposed methods, numerical examples with datasets obtained by numerical material tests on representative volume elements are carried out. Validation of the results is performed using both test load cases from the numerical dataset, as well as application as a constitutive model in the finite element method.

Collision Mitigation by Dolphin Ant Lion Optimizer Pre-Packet Scheduling Approach

A wireless sensor network (WSN) consists of distributed autonomous sensors deployed over a large geographical area. Wireless Sensor Networks (WSNs) are composed of large-scale sensors that are specifically assigned to do particular tasks, with a significant portion of these tasks involving reporting and monitoring activities. Nevertheless, because to the potential expansion of the network to several sensor nodes, the likelihood of collision is significantly increased. This research presents an innovative approach for conducting collision detection and mitigation in wireless sensor networks (WSN). The first step involves conducting a simulation of Wireless Sensor Networks (WSN), followed by the utilization of the Fractional Artificial Bee Colony (FABC) algorithm for the selection of cluster heads. In this context, the network-based parameter is derived by considering factors such as the Received Signal Strength Index (RSSI), priority level, delivery rate, and energy consumption. The Deep Recurrent Neural Network (DRNN) has been modified to suit the task of collision detection. The training of the deep recurrent neural network (DRNN) is conducted via the Lion Crow Search optimizer (LCSO). Following the completion of collision detection, the subsequent step involves the implementation of a collision mitigation process utilizing a pre-scheduling technique known as Dolphin Ant Lion Optimizer (Dolphin ALO). In this context, the evaluation of fitness encompasses many factors related to collision mitigation, including energy consumption, Sleep Index (SI), delivery rate, priority level, E-waste management, and E-save measures. The approach presented in this study demonstrated superior performance in terms of energy consumption, throughput, Packet Delivery Ratio (PDR), and collision detection rate. Specifically, it achieved the least energy consumption of 0.185, the greatest throughput of 0.815, the highest PDR of 0.815, and the highest collision detection rate of 0.930.

A Lightweight Reconstruction Model via a Neural Network for a Video Super-Resolution Model

Super-resolution in image and video processing has been a challenge in computer vision, with its progression creating substantial societal ramifications. More specifically, video super-resolution methodologies aim to restore spatial details while upholding the temporal coherence among frames. Nevertheless, their extensive parameter counts and high demand for computational resources challenge the deployment of existing deep convolutional neural networks on mobile platforms. In response to these concerns, our research undertakes an in-depth investigation into deep convolutional neural networks and offers a lightweight model for video super-resolution, capable of reducing computational load. In this study, we bring forward a unique lightweight model for video super-resolution, the Deep Residual Recursive Network (DRRN). The model applies residual learning to stabilize the Recurrent Neural Network (RNN) training, meanwhile adopting depth-wise separable convolution to boost the efficiency of super-resolution operations. Thorough experimental evaluations reveal that our proposed model excels in computational efficiency and in generating refined and temporally consistent results for video super-resolution. Hence, this research presents a crucial stride toward applying video super-resolution strategies on devices with resource limitations.

A layer-based random SGD for distributed training of RNNs

Since frequent data communication leads to increased time consumption in distributed training, compression of communication data is one of the effective ways to speed up the whole training process. Many existing compression methods introduce extra time overhead in data compression and decompression, for instance, Top-K and Random-k. There are also methods that can reduce the extra time overhead but suffer from the shortcoming of insufficient generalization. For example, the layer-based compression LR-SGD (Layer-based Random Stochastic Gradient Descent) cannot be applied to the training of recurrent neural networks. In this paper, an improved method based on LR-SGD is proposed. By being divided into virtual layers for compression, this improved method is able to train various networks efficiently. To validate the proposed method, experiments were conducted using different neural networks on a stimulative cluster. The results demonstrate that the proposed method applies well on different networks with low communication overhead and a nice convergence effect.

Recurrent Neural Network Training With Convex Loss and Regularization Functions by Extended Kalman Filtering

This paper investigates the use of extended Kalman filtering to train recurrent neural networks with rather general convex loss functions and regularization terms on the network parameters, including <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell _{1}$</tex-math></inline-formula> -regularization. We show that the learning method is competitive with respect to stochastic gradient descent in a nonlinear system identification benchmark and in training a linear system with binary outputs. We also explore the use of the algorithm in data-driven nonlinear model predictive control and its relation with disturbance models for offset-free closed-loop tracking.

Predicting human poses via recurrent attention network

Human motion prediction is a challenging task due to the diversity and randomness of future poses. Due to the inherent topology of pose data, most recent work has used graph convolution networks (GCNs) to accomplish the task of human motion prediction. However, the GCN-based method has a major shortcoming in that the modeling of temporal information is insufficient. In this paper, we propose a simple approach that combines recurrent neural networks (RNNs) and attention mechanism for motion prediction, which considers both spatial relations between different joints and temporal correlation. The query, key and value of the attention mechanism allow us to select the information we need for the subsequent prediction. To solve the difficult problem of RNN training, we utilize uncertainty and strong short-term constraints to optimize the training process. We evaluate our method on several standard benchmark datasets for human motion prediction, i.e., the Human3.6M dataset and the CMU MoCap dataset. The experimental results show that our approach outperforms previous approaches.

Mining the contribution of intensive care clinical course to outcome after traumatic brain injury

Existing methods to characterise the evolving condition of traumatic brain injury (TBI) patients in the intensive care unit (ICU) do not capture the context necessary for individualising treatment. Here, we integrate all heterogenous data stored in medical records (1166 pre-ICU and ICU variables) to model the individualised contribution of clinical course to 6-month functional outcome on the Glasgow Outcome Scale -Extended (GOSE). On a prospective cohort (n = 1550, 65 centres) of TBI patients, we train recurrent neural network models to map a token-embedded time series representation of all variables (including missing values) to an ordinal GOSE prognosis every 2 h. The full range of variables explains up to 52% (95% CI: 50–54%) of the ordinal variance in functional outcome. Up to 91% (95% CI: 90–91%) of this explanation is derived from pre-ICU and admission information (i.e., static variables). Information collected in the ICU (i.e., dynamic variables) increases explanation (by up to 5% [95% CI: 4–6%]), though not enough to counter poorer overall performance in longer-stay (>5.75 days) patients. Highest-contributing variables include physician-based prognoses, CT features, and markers of neurological function. Whilst static information currently accounts for the majority of functional outcome explanation after TBI, data-driven analysis highlights investigative avenues to improve the dynamic characterisation of longer-stay patients. Moreover, our modelling strategy proves useful for converting large patient records into interpretable time series with missing data integration and minimal processing.

Training RNN language models on uncertain ASR hypotheses in limited data scenarios

Training domain-specific automatic speech recognition (ASR) systems requires a suitable amount of data comprising the target domain. In several scenarios, such as early development stages, privacy-critical applications, or under-resourced languages, only a limited amount of in-domain speech data and an even smaller amount of manual text transcriptions, if any, are available. This motivates the study of ASR language model (LM) training on a limited amount of in-domain speech data. Early works have attempted training of n-gram LMs from ASR N-best lists and lattices but training and adaptation of recurrent neural network (RNN) LMs from ASR transcripts has not received attention. In this work, we study training and adaptation of RNN LMs using alternate, uncertain ASR hypotheses embedded in ASR confusion networks obtained from target domain speech data. We explore different methods for training the RNN LMs to deal with the uncertain input sequences. The first method extends the cross-entropy objective into a Kullback–Leibler (KL) divergence based training loss, the second method formulates a training loss based on a hidden Markov model (HMM), and the third method performs training on paths sampled from the confusion networks. These methods are applied to limited data setups including telephone and meeting conversation datasets. Performance is evaluated under two settings wherein no manual transcriptions or a small amount of manual transcriptions are available to aid the training. Moreover, a model adaptation setting is also evaluated wherein the RNN LM is pre-trained on an out-of-domain conversational corpus. Overall the sampling method for training RNN LMs on ASR confusion networks performs the best, and results in up to 12% relative reduction in perplexity on the meeting dataset as compared to training on ASR 1-best hypotheses, without any manual transcriptions. However, the perplexity reductions do not translate into equivalent WER reductions. A detailed analysis of the perplexity reductions obtained by the different methods is performed in order to understand this effect.

Warming up recurrent neural networks to maximise reachable multistability greatly improves learning

Training recurrent neural networks is known to be difficult when time dependencies become long. In this work, we show that most standard cells only have one stable equilibrium at initialisation, and that learning on tasks with long time dependencies generally occurs once the number of network stable equilibria increases; a property known as multistability. Multistability is often not easily attained by initially monostable networks, making learning of long time dependencies between inputs and outputs difficult. This insight leads to the design of a novel way to initialise any recurrent cell connectivity through a procedure called “warmup” to improve its capability to learn arbitrarily long time dependencies. This initialisation procedure is designed to maximise network reachable multistability, i.e., the number of equilibria within the network that can be reached through relevant input trajectories, in few gradient steps. We show on several information restitution, sequence classification, and reinforcement learning benchmarks that warming up greatly improves learning speed and performance, for multiple recurrent cells, but sometimes impedes precision. We therefore introduce a double-layer architecture initialised with a partial warmup that is shown to greatly improve learning of long time dependencies while maintaining high levels of precision. This approach provides a general framework for improving learning abilities of any recurrent cell when long time dependencies are present. We also show empirically that other initialisation and pretraining procedures from the literature implicitly foster reachable multistability of recurrent cells.