Training Recurrent Neural Networks Research Articles

SCSQ: A sample cooperation optimization method with sample quality for recurrent neural networks

Time series forecasting holds significant value across various application scenarios. However, practical implementations often encounter challenges related to low-quality time series data resulting from system failures or external interference. Unfortunately, existing forecasting methods primarily focus on optimizing model architecture for accurate predictions and overlook the importance of addressing data quality issues. In this paper, we propose optimizing data utilization to enhance model performance based on data quality and introduce the Sample Cooperation with Sample Quality (SCSQ) method to facilitate recurrent neural networks training. Firstly, we define sample quality as the matching degree between samples and model, and suggest using the attention entropy to calculate the sample quality through an attention mechanism. Secondly, we optimize the model's gradient vector based on sample quality. To address optimization training involving samples of different qualities effectively, we propose a more reasonable objective function within our proposed sample gradient conflict optimization module and devise a novel algorithm leveraging approximation techniques along with ADMM algorithm implementation. Through experiments conducted on six datasets, the results demonstrate that SCSQ significantly improves LSTM's performance. In certain cases, it even surpasses the state-of-the-art models. Additionally, SCSQ enhances the anti-interference ability of LSTM against low-quality samples.

Liquid-Liquid Dispersion Performance Prediction and Uncertainty Quantification Using Recurrent Neural Networks.

We demonstrate the application of a recurrent neural network (RNN) to perform multistep and multivariate time-series performance predictions for stirred and static mixers as exemplars of complex multiphase systems. We employ two network architectures in this study, fitted with either long short-term memory and gated recurrent unit cells, which are trained on high-fidelity, three-dimensional, computational fluid dynamics simulations of the mixer performance, in the presence and absence of surfactants, in terms of drop size distributions and interfacial areas as a function of system parameters; these include physicochemical properties, mixer geometry, and operating conditions. Our results demonstrate that while it is possible to train RNNs with a single fully connected layer more efficiently than with an encoder-decoder structure, the latter is shown to be more capable of learning long-term dynamics underlying dispersion metrics. Details of the methodology are presented, which include data preprocessing, RNN model exploration, and methods for model performance visualization; an ensemble-based procedure is also introduced to provide a measure of the model uncertainty. The workflow is designed to be generic and can be deployed to make predictions in other industrial applications with similar time-series data.

Exploring Flip Flop memories and beyond: training Recurrent Neural Networks with key insights.

Training neural networks to perform different tasks is relevant across various disciplines. In particular, Recurrent Neural Networks (RNNs) are of great interest in Computational Neuroscience. Open-source frameworks dedicated to Machine Learning, such as Tensorflow and Keras have produced significant changes in the development of technologies that we currently use. This work contributes by comprehensively investigating and describing the application of RNNs for temporal processing through a study of a 3-bit Flip Flop memory implementation. We delve into the entire modeling process, encompassing equations, task parametrization, and software development. The obtained networks are meticulously analyzed to elucidate dynamics, aided by an array of visualization and analysis tools. Moreover, the provided code is versatile enough to facilitate the modeling of diverse tasks and systems. Furthermore, we present how memory states can be efficiently stored in the vertices of a cube in the dimensionally reduced space, supplementing previous results with a distinct approach.

Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training

Recurrent Neural Networks (RNNs) are useful in temporal sequence tasks. However, training RNNs involves dense matrix multiplications which require hardware that can support a large number of arithmetic operations and memory accesses. Implementing online training of RNNs on the edge calls for optimized algorithms for an efficient deployment on hardware. Inspired by the spiking neuron model, the Delta RNN exploits temporal sparsity during inference by skipping over the update of hidden states from those inactivated neurons whose change of activation across two timesteps is below a defined threshold. This work describes a training algorithm for Delta RNNs that exploits temporal sparsity in the backward propagation phase to reduce computational requirements for training on the edge. Due to the symmetric computation graphs of forward and backward propagation during training, the gradient computation of inactivated neurons can be skipped. Results show a reduction of ∼80% in matrix operations for training a 56k parameter Delta LSTM on the Fluent Speech Commands dataset with negligible accuracy loss. Logic simulations of a hardware accelerator designed for the training algorithm show 2-10X speedup in matrix computations for an activation sparsity range of 50%-90%. Additionally, we show that the proposed Delta RNN training will be useful for online incremental learning on edge devices with limited computing resources.

Analysis of a Two-Step Gradient Method with Two Momentum Parameters for Strongly Convex Unconstrained Optimization

The paper is devoted to the theoretical and numerical analysis of the two-step method, constructed as a modification of Polyak’s heavy ball method with the inclusion of an additional momentum parameter. For the quadratic case, the convergence conditions are obtained with the use of the first Lyapunov method. For the non-quadratic case, sufficiently smooth strongly convex functions are obtained, and these conditions guarantee local convergence.An approach to finding optimal parameter values based on the solution of a constrained optimization problem is proposed. The effect of an additional parameter on the convergence rate is analyzed. With the use of an ordinary differential equation, equivalent to the method, the damping effect of this parameter on the oscillations, which is typical for the non-monotonic convergence of the heavy ball method, is demonstrated. In different numerical examples for non-quadratic convex and non-convex test functions and machine learning problems (regularized smoothed elastic net regression, logistic regression, and recurrent neural network training), the positive influence of an additional parameter value on the convergence process is demonstrated.

Subcutaneous temperature prediction during cryogenic jet cooling by surface temperature measurements and RNNs

Accurate information of subcutaneous temperature is crucial to the effectiveness and safety of a variety of medical procedures by energy-based devices, such as cooling treatments. Numerous strategies for invasive subcutaneous temperature monitoring have been investigated; however, they are impractical mainly due to their requirements for precise heat transfer modeling. In the case of cooling treatment by cryogenic substances, heat transfer rate largely varies with their phase change and procedure condition, which prevents accurate prediction of subcutaneous temperature. In this work, we propose a non-invasive, accurate, and practical prediction of subcutaneous temperature during CO2 spray jet cooling without numerical modeling by utilizing recurrent neural networks (RNNs) with surface temperature measurements for the first time. A patchable temperature sensor array on a flexible PCB is used to measure surface temperature. With this measured surface temperature, RNNs are developed with two functionalities: extraction of thermophysical properties (thermal conductivity and thermal diffusivity) of the target substrate and prediction of temperature response at 1-mm depth from the surface, which is the typical location of pain points. Five polymer thin films are used with the reference thermocouple embedded inside to train RNNs. A porcine skin is used to validate the RNNs, which show a prediction accuracy of 99.2%.

Breaking Time Invariance: Assorted-Time Normalization for RNNs

Methods such as Layer Normalization (LN) and Batch Normalization have proven to be effective in improving the training of Recurrent Neural Networks (RNNs). However, existing methods normalize using only the instantaneous information at one particular time step, and the result of the normalization is a preactivation state with a time-independent distribution. This implementation fails to account for certain temporal differences inherent in the inputs and the architecture of RNNs. Since these networks share weights across time steps, it may also be desirable to account for the connections between time steps in the normalization scheme. In this paper, we propose a normalization method called Assorted-Time Normalization (ATN), which preserves information from multiple consecutive time steps and normalizes using them. This setup allows us to introduce longer time dependencies into the traditional normalization methods without introducing any new trainable parameters. We present theoretical derivations for the gradient propagation and prove the weight scaling invariance property. Our experiments applying ATN to LN demonstrate consistent improvement on various tasks, such as Adding, Copying, and Denoise Problems and Language Modeling Problems.

ALGORITHM FOR DETECTION OF ABUSIVE CONTENT IN AUDIO CONTENT FOR IMPLEMENTATION IN OBJECT-ORIENTED INFORMATION SYSTEM

The paper proposes the basic principles of developing an object-oriented information system for detecting abusive content in Ukrainian-language audio content based on a new algorithm that uses statistical and neural network approaches to detect abusive content. Detecting offensive content in text and audio content is an urgent task, as it helps to create a safe and healthy environment for communication, especially in online platforms. Offensive content can harm the people who hear or read it and violate their rights. It can also have a negative impact on society, contributing to the spread of hatred and violence. To detect abusive speech in audio content, the proposed approach uses two key components: the use of dictionary methods and the analysis of the emotional tonality of utterances. A set of reviews was used as a dataset to determine the abusive component of the content, which was expanded by the authors by adding words of abuse. An object-oriented information system architecture written in the Python programming language in the PyCharm programming environment is proposed. The information system consists of a software module for training recurrent neural network models and further saving trained instances, and a software module for detecting abusive content in Ukrainian-language audio content using trained RNN models. Since the recurrent neural network is trained on a short text data set, the system is less efficient at identifying texts that have a larger number of words. In the example of the proposed approach, the accuracy of detecting offensive content is more than 90%. This means that the algorithm works correctly in the absence of offending highlights in the test data set. The results of the analysis of the effectiveness of the proposed approach show that in the vast majority of cases the conclusions regarding the acceptability of audio content based on the level of abuse are correct.

Topological features of spike trains in recurrent spiking neural networks that are trained to generate spatiotemporal patterns.

In this study, we focus on training recurrent spiking neural networks to generate spatiotemporal patterns in the form of closed two-dimensional trajectories. Spike trains in the trained networks are examined in terms of their dissimilarity using the Victor-Purpura distance. We apply algebraic topology methods to the matrices obtained by rank-ordering the entries of the distance matrices, specifically calculating the persistence barcodes and Betti curves. By comparing the features of different types of output patterns, we uncover the complex relations between low-dimensional target signals and the underlying multidimensional spike trains.

Leakage monitoring and diagnosis of LNG storage tanks with temperature sensing network integration and artificial intelligence algorithm

Temperature is the most important safety monitoring indicator for leakage diagnosis during the operation phase of liquefied natural gas (LNG) storage tanks. Timely monitoring and accurate identification of LNG leakage events are crucial for accident prevention, loss reduction, and facility safety maintenance. This study integrates artificial intelligence (AI) algorithms and temperature sensing data to achieve intelligent monitoring and diagnosis of leakage in LNG storage tanks. Firstly, a comprehensive temperature sensing network is constructed by combining numerical simulation of the temperature field and temperature sensing experiments using fiber Bragg grating sensors. Secondly, Python is used to perform linear grid interpolation and flattening on the sensing network, generating 2D temperature nephogram samples that are conducive to AI algorithm recognition. Finally, sample features are extracted using machine vision, and leakage location calculation, leakage diagnosis and leakage volume calculation are implemented with the help of machine learning algorithms, achieving satisfactory accuracy on the test set. In addition, the ConvLSTM framework is introduced for deep learning and recurrent neural network training, enabling spatiotemporal prediction of the leakage area.

Trained recurrent neural networks develop phase-locked limit cycles in a working memory task.

Neural oscillations are ubiquitously observed in many brain areas. One proposed functional role of these oscillations is that they serve as an internal clock, or 'frame of reference'. Information can be encoded by the timing of neural activity relative to the phase of such oscillations. In line with this hypothesis, there have been multiple empirical observations of such phase codes in the brain. Here we ask: What kind of neural dynamics support phase coding of information with neural oscillations? We tackled this question by analyzing recurrent neural networks (RNNs) that were trained on a working memory task. The networks were given access to an external reference oscillation and tasked to produce an oscillation, such that the phase difference between the reference and output oscillation maintains the identity of transient stimuli. We found that networks converged to stable oscillatory dynamics. Reverse engineering these networks revealed that each phase-coded memory corresponds to a separate limit cycle attractor. We characterized how the stability of the attractor dynamics depends on both reference oscillation amplitude and frequency, properties that can be experimentally observed. To understand the connectivity structures that underlie these dynamics, we showed that trained networks can be described as two phase-coupled oscillators. Using this insight, we condensed our trained networks to a reduced model consisting of two functional modules: One that generates an oscillation and one that implements a coupling function between the internal oscillation and external reference. In summary, by reverse engineering the dynamics and connectivity of trained RNNs, we propose a mechanism by which neural networks can harness reference oscillations for working memory. Specifically, we propose that a phase-coding network generates autonomous oscillations which it couples to an external reference oscillation in a multi-stable fashion.

Brain control of bimanual movement enabled by recurrent neural networks

Brain-computer interfaces have so far focused largely on enabling the control of a single effector, for example a single computer cursor or robotic arm. Restoring multi-effector motion could unlock greater functionality for people with paralysis (e.g., bimanual movement). However, it may prove challenging to decode the simultaneous motion of multiple effectors, as we recently found that a compositional neural code links movements across all limbs and that neural tuning changes nonlinearly during dual-effector motion. Here, we demonstrate the feasibility of high-quality bimanual control of two cursors via neural network (NN) decoders. Through simulations, we show that NNs leverage a neural ‘laterality’ dimension to distinguish between left and right-hand movements as neural tuning to both hands become increasingly correlated. In training recurrent neural networks (RNNs) for two-cursor control, we developed a method that alters the temporal structure of the training data by dilating/compressing it in time and re-ordering it, which we show helps RNNs successfully generalize to the online setting. With this method, we demonstrate that a person with paralysis can control two computer cursors simultaneously. Our results suggest that neural network decoders may be advantageous for multi-effector decoding, provided they are designed to transfer to the online setting.

SFR Modeling for Hybrid Power Systems Based on Deep Transfer Learning

A deep transfer learning method is presented for establishing the aggregated system frequency response (SFR) model of wind-thermal hybrid power systems (HPSs). In order to deal with nonlinearities and non-Gaussian disturbances, the quadratic survival information potential (QSIP) of the squared identification error is employed to construct the performance index when training recurrent neural networks (RNNs). A pre-trained SFR model is then obtained by the improved RNNs using the source domain data collected from the HPS in historical scenarios. Subsequently, the maximum mean difference is utilized to test the similarity of the HPS in historical and current scenarios. After that, the pre-trained SFR model is fine-tuned by adding some nodes to the recurrent layer and a functional link to the input layer. The SFR model of the HPS operating in current scenario can then be built based on the transferred source domain pre-trained SFR model. Simulation results illustrate that the proposed data driven modelling method can obtain accurate, effective and timely SFR model for a wind-thermal HPS with different wind speeds and load disturbances.

Enhancing bearing fault diagnosis using motor current signals: A novel approach combining time shifting and CausalConvNets

In motor drive system, Bearing fault detection through motor current signal (MCS) analysis has gained recognition for its cost-effectiveness and non-invasive nature. However, two-dimensional (2D) convolutional neural networks (CNNs) exhibit suboptimal performance when directly applied to raw stator current signals for fault detection. Meanwhile, the training of Recurrent Neural Networks (RNNs) for processing one-dimensional (1D) temporal sequence can be a time-consuming task due to the inherent procedures of the network structure. To balance time-consuming and accuracy, this paper presents a novel MCS-based bearing fault detection model utilizing a 1D CNN and corresponding pre-processing methods. In the early stages, a noise cancellation strategy termed time shifting is implemented. Subsequently, wavelet transform is employed to further refine the processed signal by eliminating components with minimal significance. By compressing homogenous signals, the resultant tensor, referred to as the “root map”, is employed as the input for the proposed temporal causal network. Two groups of experimental results conducted on both laboratory and industrial conditions validate the effectiveness and convenience of the proposed methodology.

Toward Effective Aircraft Call Sign Detection Using Fuzzy String-Matching between ASR and ADS-B Data

Recently, artificial intelligence and data science have witnessed dramatic progress and rapid growth, especially Automatic Speech Recognition (ASR) technology based on Hidden Markov Models (HMMs) and Deep Neural Networks (DNNs). Consequently, new end-to-end Recurrent Neural Network (RNN) toolkits were developed with higher speed and accuracy that can often achieve a Word Error Rate (WER) below 10%. These toolkits can nowadays be deployed, for instance, within aircraft cockpits and Air Traffic Control (ATC) systems in order to identify aircraft and display recognized voice messages related to flight data, especially for airports not equipped with radar. Hence, the performance of air traffic controllers and pilots can ultimately be improved by reducing workload and stress and enforcing safety standards. Our experiment conducted at Tangier’s International Airport ATC aimed to build an ASR model that is able to recognize aircraft call signs in a fast and accurate way. The acoustic and linguistic models were trained on the Ibn Battouta Speech Corpus (IBSC), resulting in an unprecedented speech dataset with approved transcription that includes real weather aerodrome observation data and flight information with a call sign captured by an ADS-B receiver. All of these data were synchronized with voice recordings in a structured format. We calculated the WER to evaluate the model’s accuracy and compared different methods of dataset training for model building and adaptation. Despite the high interference in the VHF radio communication channel and fast-speaking conditions that increased the WER level to 20%, our standalone and low-cost ASR system with a trained RNN model, supported by the Deep Speech toolkit, was able to achieve call sign detection rate scores up to 96% in air traffic controller messages and 90% in pilot messages while displaying related flight information from ADS-B data using the Fuzzy string-matching algorithm.

Statistical physics of learning in high-dimensional chaotic systems

In many complex systems, elementary units live in a chaotic environment and need to adapt their strategies to perform a task by extracting information from the environment and controlling the feedback loop on it. One of the main examples of systems of this kind is provided by recurrent neural networks. In this case, recurrent connections between neurons drive chaotic behavior, and when learning takes place, the response of the system to a perturbation should also take into account its feedback on the dynamics of the network itself. In this work, we consider an abstract model of a high-dimensional chaotic system as a paradigmatic model and study its dynamics. We study the model under two particular settings: Hebbian driving and FORCE training. In the first case, we show that Hebbian driving can be used to tune the level of chaos in the dynamics, and this reproduces some results recently obtained in the study of more biologically realistic models of recurrent neural networks. In the latter case, we show that the dynamical system can be trained to reproduce simple periodic functions. To do this, we consider the FORCE algorithm—originally developed to train recurrent neural networks—and adapt it to our high-dimensional chaotic system. We show that this algorithm drives the dynamics close to an asymptotic attractor the larger the training time. All our results are valid in the thermodynamic limit due to an exact analysis of the dynamics through dynamical mean field theory.

Research of Carbon Emission Prediction: An Oscillatory Particle Swarm Optimization for Long Short-Term Memory

Carbon emissions play a significant role in shaping social policy-making, industrial planning, and other critical areas. Recurrent neural networks (RNNs) serve as the major choice for carbon emission prediction. However, year-frequency carbon emission data always results in overfitting during RNN training. To address this issue, we propose a novel model that combines oscillatory particle swarm optimization (OPSO) with long short-term memory (LSTM). OPSO is employed to fine-tune the hyperparameters of LSTM, utilizing an oscillatory strategy to effectively mitigate overfitting and consequently improve the accuracy of the LSTM model. In validation tests, real data from Hainan Province, encompassing diverse dimensions such as gross domestic product, forest area, and ten other relevant factors, are used. Standard LSTM and PSO-LSTM are selected in the control group. The mean absolute error (MAE), root mean square error (RMSE), and mean absolute percentage error (MAPE) are used to evaluate the performance of these methods. In the test dataset, the MAE of OPSO-LSTM is 117.708, 65.72% better than LSTM and 29.48% better than PSO-LSTM. The RMSE of OPSO-LSTM is 149.939, 68.52% better than LSTM and 41.90% better than PSO-LSTM. The MAPE of OPSO-LSTM is 0.017, 65.31% better than LSTM, 29.17% better than PSO-LSTM. The experimental results prove that OPSO-LSTM can provide reliable predictions for carbon emissions.

Privacy-Preserving Federated Recurrent Neural Networks

We present RHODE, a novel system that enables privacy-preserving training of and prediction on Recurrent Neural Networks (RNNs) in a cross-silo federated learning setting by relying on multiparty homomorphic encryption. RHODE preserves the confidentiality of the training data, the model, and the prediction data; and it mitigates federated learning attacks that target the gradients under a passive-adversary threat model. We propose a packing scheme, multi-dimensional packing, for a better utilization of Single Instruction, Multiple Data (SIMD) operations under encryption. With multi-dimensional packing, RHODE enables the efficient processing, in parallel, of a batch of samples. To avoid the exploding gradients problem, RHODE provides several clipping approximations for performing gradient clipping under encryption. We experimentally show that the model performance with RHODE remains similar to non-secure solutions both for homogeneous and heterogeneous data distributions among the data holders. Our experimental evaluation shows that RHODE scales linearly with the number of data holders and the number of timesteps, sub-linearly and sub-quadratically with the number of features and the number of hidden units of RNNs, respectively. To the best of our knowledge, RHODE is the first system that provides the building blocks for the training of RNNs and its variants, under encryption in a federated learning setting.

A High-Security Probabilistic Constellation Shaping Transmission Scheme Based on Recurrent Neural Networks

In this paper, a high-security probabilistic constellation shaping transmission scheme based on recurrent neural networks (RNNs) is proposed, in which the constellation point probabilistic distribution is generated based on recurrent neural network training. A 4D biplane fractional-order chaotic system is introduced to ensure the security performance of the system. The performance of the proposed scheme is verified in a 2 km seven-core optical transmission system. The RNN-trained probabilistic shaping scheme achieves a transmission gain of 1.23 dB compared to the standard 16QAM signal, 0.39 dB compared to the standard Maxwell-Boltzmann (M-B) distribution signal, and a higher net bit rate. The proposed encryption scheme has higher randomness and security than the conventional integer-order chaotic system, with a key space of 10,163. This scheme will have a promising future fiber optic transmission scheme because it combines the efficient transmission and security of fiber optic transmission systems.

Continuous self‐adversarial training of recurrent neural network–based constitutive description

AbstractData‐driven methods yield advantages in computational homogenization approaches due to the ability to capture complex material behaviour without the necessity to assume specific constitutive models. Neural network–based constitutive descriptions are one of the most widely used data‐driven approaches in the context of computational mechanics. The accuracy of this method strongly depends on the available data. Additionally, when considering inelastic materials, whose constitutive responses depend on the loading history, the accuracy and robustness of the approximation are influenced by the training algorithm. The applied recurrent neural networks exhibit reduced robustness in the presence of errors in the input. When capturing the history dependency using previously predicted material responses, occurrences of prediction errors accumulate over several time steps. An approach for achieving enhanced robustness of the predictions is based on extending the initial training dataset by iteratively generating adversarial examples, subjected to perturbations, based on the current prediction errors. In this contribution, a continuous self‐adversarial training approach yielding robust recurrent neural network constitutive descriptions for inelastic materials is presented. Compared to the iterative method it is based on, it exhibits significantly improved training efficiency. In order to demonstrate the capabilities of the proposed methods, numerical examples with datasets obtained by numerical material tests on representative volume elements are carried out. Validation of the results is performed using both test load cases from the numerical dataset, as well as application as a constitutive model in the finite element method.