Training Recurrent Neural Networks Research Articles

Robustness analysis and training of recurrent neural networks using dissipativity theory

Abstract Neural networks are widely applied in control applications, yet providing safety guarantees for neural networks is challenging due to their highly nonlinear nature. We provide a comprehensive introduction to the analysis of recurrent neural networks (RNNs) using robust control and dissipativity theory. Specifically, we consider H 2 {\mathcal{H}_{2}} -performance and the ℓ 2 {\ell _{2}} -gain to quantify the robustness of dynamic RNNs with respect to input perturbations. First, we analyze the robustness of RNNs using the proposed robustness certificates and then, we present linear matrix inequality constraints to be used in training of RNNs to enforce robustness. Finally, we illustrate in a numerical example that the proposed approach enhances the robustness of RNNs.

A spatio-temporal sequence-to-sequence network for traffic flow prediction

Spatio-temporal prediction has drawn much attention given its wide application, of which traffic flow prediction is a typical task. Within the vision of smart cities, traffic flow prediction plays a vital role in traffic control and optimization. The current approaches commonly use a graph convolutional network (GCN) to capture any spatial correlations and a recurrent neural network (RNN) to mine any temporal correlations. However, GCNs cannot detect spatial heterogeneity and time-varying spatial correlations, and RNNs cannot model the periodicity of traffic series data. Further, iterative training of RNNs may come at a high computational cost and result in problems with error propagation. To this end, we propose STSSN, a spatio-temporal sequence-to-sequence network, that not only explores heterogeneous and time-varying spatial correlations, but also efficiently exploits sequential and periodic temporal correlations. STSSN is based on an encoder-decoder framework. In the network, the model’s input is processed to extract the periodic daily and weekly patterns in traffic flows. Both the encoder and decoder mainly consist of an enhanced diffusion convolutional network (EDCN) and a temporal convolutional network (TCN). In the EDCN module, the diffusion convolution incorporates time-varying node representations so as to capture both node-specific patterns and time-varying spatial correlations. In the TCN module, we take full advantage of the parallel computing in the dilated causal convolution to mine local (short-term) temporal correlations. More importantly, global (long-term) temporal correlations are discovered through an encoder-decoder attention (EDA) module. This EDA mechanism directly models the relationship between the encoder and decoder to mitigate problems with error propagation. Experiments on two real-world datasets verify the superiority of STSSN, with STSSN’s MAE at between 3.85%-6.17% lower than the state-of-the-art baselines on the PEMS-BAY dataset.

A Reinforcement Learning Approach for Rebalancing Electric Vehicle Sharing Systems

This paper proposes a reinforcement learning approach for nightly offline rebalancing operations in free-floating electric vehicle sharing systems (FFEVSS). Due to sparse demand in a network, FFEVSS requires relocation of electrical vehicles (EVs) to charging stations and demander nodes, which is typically done by a group of drivers. A shuttle is used to pick up and drop off drivers throughout the network. The objective of this study is to solve the shuttle routing problem to finish the rebalancing work in minimal time. We consider a reinforcement learning framework for the problem, in which a central controller determines the routing policies of a fleet of multiple shuttles. We deploy a policy gradient method for training recurrent neural networks and compare the obtained policy results with heuristic solutions. Our numerical studies show that unlike the existing solutions in the literature, the proposed methods allow solving the general version of the problem with no restrictions on the urban EV network structure and charging requirements of EVs. Moreover, the learned policies offer a wide range of flexibility, resulting in a significant reduction in the time needed to rebalance the network.

FARNN: FPGA-GPU Hybrid Acceleration Platform for Recurrent Neural Networks

GPU-based platforms provide high computation throughput for large mini-batch deep neural network computations. However, a large batch size may not be ideal for some situations, such as aiming at low latency, training on edge/mobile devices, partial retraining for personalization, and having irregular input sequence lengths. GPU performance suffers from low utilization especially for small-batch recurrent neural network (RNN) applications where sequential computations are required. In this article, we propose a hybrid architecture, called FARNN, which combines a GPU and an FPGA to accelerate RNN computation for small batch sizes. After separating RNN computations into GPU-efficient and GPU-inefficient tasks, we design special FPGA computation units that accelerate the GPU-inefficient RNN tasks. FARNN off-loads the GPU-inefficient tasks to the FPGA. We evaluate FARNN with synthetic RNN layers of various configurations on the Xilinx UltraScale+ FPGA and the NVIDIA P100 GPU in addition to evaluating it with real RNN applications. The evaluation result indicates that FARNN outperforms the P100 GPU platform for RNN training by up to 4.2 <inline-formula><tex-math notation="LaTeX">$\times {}$</tex-math></inline-formula> with small batch sizes, long input sequences, and many RNN cells per layer.

Application of deep learning in the detection and practical evaluation of ski resort teaching area

SummaryThe teaching land of ski resorts provides learning sites for beginners of skiing. The teaching sites of different ski resorts are different. When selecting teaching scenes, we need to focus on the safety of the site. The focus of our research is to detect more suitable teaching land in the whole ski resort and evaluate its practical indicators. In order to accurately detect the teaching area in the snow field, we propose a method based on deep learning. The algorithm innovatively optimizes the convolution network. The experiment combines convolutional neural network with recurrent neural network for training, and uses the output of convolutional neural network to train recurrent neural network, so as to continuously predict the next optimal teaching area in the sequence. The purpose of training is to optimize the detection result area. The experimental results show that the model can better detect the sites suitable for teaching in the snow field.

Coordinate Descent on the Orthogonal Group for Recurrent Neural Network Training

We address the poor scalability of learning algorithms for orthogonal recurrent neural networks via the use of stochastic coordinate descent on the orthogonal group, leading to a cost per iteration that increases linearly with the number of recurrent states. This contrasts with the cubic dependency of typical feasible algorithms such as stochastic Riemannian gradient descent, which prohibits the use of big network architectures. Coordinate descent rotates successively two columns of the recurrent matrix. When the coordinate (i.e., indices of rotated columns) is selected uniformly at random at each iteration, we prove convergence of the algorithm under standard assumptions on the loss function, stepsize and minibatch noise. In addition, we numerically show that the Riemannian gradient has an approximately sparse structure. Leveraging this observation, we propose a variant of our proposed algorithm that relies on the Gauss-Southwell coordinate selection rule. Experiments on a benchmark recurrent neural network training problem show that the proposed approach is a very promising step towards the training of orthogonal recurrent neural networks with big architectures.

CS-RNN: efficient training of recurrent neural networks with continuous skips

CS-RNN: efficient training of recurrent neural networks with continuous skips

RECURRENT NEURAL NETWORK FOR MONITORING MOUSE EMBRYONIC STEM CELL COLONY IN VITRO USING TIME-LAPSE FLUORESCENCE MICROSCOPY IMAGES

Embryonic stem cells represent a cellular resource for basic biological studies and used as medically relevant cells in vitro. Time-lapse fluorescence microscopy images taken during cell culture are frequently used to monitor time-series morphology changes and status transitions of embryonic stem cell (ESC) colonies, and to select culture medium for maintaining ESCs in undifferentiated or early differentiated stages. Recurrent Neural Network (RNN)-based prediction of time-lapse images generated in future culture can be an effective method for monitoring. Because status transitions of ESC colonies are usually complete in a few consecutive frames of time-lapse images, this study proposes a Long Short-Term Memory (LSTM) structure that incorporates a few new images taken from incubators to learn the newest colony morphology and status for a new RNN training and thus generate predicted time-lapse images. Image processing procedures of entropy, bi-lateral filter and convex-hull calculations are implemented to extract colony regions on the images. Morphology changes and status transitions are then calculated using the extracted ESC colonies. The experiment results show predicted time-lapse images in 5h future and the colony morphology and status transition calculation give accurate visualization and quantitative analyses to monitor the mouse ESCs in vitro at the undifferentiated or early differentiated stages.

On Recurrent Neural Networks for learning-based control: Recent results and ideas for future developments

This paper aims to discuss and analyze the potentialities of Recurrent Neural Networks (RNN) in control design applications. The main families of RNN are considered, namely Neural Nonlinear AutoRegressive eXogenous, Echo State Networks, Long Short Term Memory, and Gated Recurrent Units. The goal is twofold. Firstly, to survey recent results concerning the training of RNN that enjoy Input-to-State Stability (ISS) and Incremental Input-to-State Stability (δISS) guarantees. Secondly, to discuss the issues that still hinder the widespread use of RNN for control, namely their robustness, verifiability, and interpretability. The former properties are related to the so-called generalization capabilities of the networks, i.e. their consistency with the underlying real plants, even in presence of unseen or perturbed input trajectories. The latter is instead related to the possibility of providing a clear formal connection between the RNN model and the plant. In this context, we illustrate how ISS and δISS represent a significant step towards the robustness and verifiability of the RNN models, while the requirement of interpretability paves the way to the use of physics-based networks. The design of model predictive controllers with RNN as plant’s model is also briefly discussed. Lastly, some of the main topics of the paper are illustrated on a simulated chemical system.

Enhanced ECC Based Authentication Protocol in Wireless Sensor Network with DoS Mitigation

Wireless Sensors Network (WSN) face various challenges such as localization, security issues, deployment of node, clustering, and data aggregation. In addition, certain technological issues in WSN such as spatial distribution, higher energy utilization, small memory, bandwidth, and so on are also noticed. The proposed protocol comprises of four various stages such as (1) user registration, (2) remedy, (3) attack prediction, and (4) reloading. Initially, a new Modified Elliptic Curve Cryptography (ECC) model is introduced to ensure the secure communication. Moreover, the implemented approach also focuses on the attack prediction phase, where the Optimized Recurrent Neural Network (RNN) model is exploited for detecting the Denial-of-Service (DoS) attack. Here, the training of RNN is carried out by Shark Smell Optimization with Opposition Learning and Mutation Process (SSOL-MP) via tuning the optimal weights. At the end, analysis is done for validating the enhancement of the suggested model.

Quantitative Predictions of Moisture-Driven Photoemission Dynamics in Metal Halide Perovskites via Machine Learning.

Metal halide perovskite (MHP) photovoltaics may become a viable alternative to standard Si-based technologies, but the current lack of long-term stability precludes their commercial adoption. Exposure to standard operational stressors (light, temperature, bias, oxygen, and water) often instigate optical and electronic dynamics, calling for a systematic investigation into MHP photophysical processes and the development of quantitative models for their prediction. We resolve the moisture-driven light emission dynamics for both methylammonium lead tribromide and triiodide thin films as a function of relative humidity (rH). With the humidity and photoluminescence time series, we train recurrent neural networks and establish their ability to quantitatively predict the path of future light emission with 18% error over 4 h. Together, our in situ rH-PL measurements and machine learning forecasting models provide a framework for the rational design of future stable perovskite devices and, thus, a faster transition toward commercial applications.

On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools

Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters. Considering RNNs as dynamical systems, a natural way to capture stability, i.e., the growth and decay over long iterates, are the Lyapunov Exponents (LEs), which form the Lyapunov spectrum. The LEs have a bearing on stability of RNN training dynamics since forward propagation of information is related to the backward propagation of error gradients. LEs measure the asymptotic rates of expansion and contraction of non-linear system trajectories, and generalize stability analysis to the time-varying attractors structuring the non-autonomous dynamics of data-driven RNNs. As a tool to understand and exploit stability of training dynamics, the Lyapunov spectrum fills an existing gap between prescriptive mathematical approaches of limited scope and computationally-expensive empirical approaches. To leverage this tool, we implement an efficient way to compute LEs for RNNs during training, discuss the aspects specific to standard RNN architectures driven by typical sequential datasets, and show that the Lyapunov spectrum can serve as a robust readout of training stability across hyperparameters. With this exposition-oriented contribution, we hope to draw attention to this under-studied, but theoretically grounded tool for understanding training stability in RNNs.

Quantitative Analysis of DCE and DSC-MRI: From Kinetic Modeling to Deep Learning.

Perfusion MRI is a well-established imaging modality with a multitude of applications in oncological and cardiovascular imaging. Clinically used processing methods, while stable and robust, have remained largely unchanged in recent years. Despite promising results from novel methods, their relatively minimal improvement compared to established methods did not generally warrant significant changes to clinical perfusion processing. Machine learning in general and deep learning in particular, which are currently revolutionizing computer-aided diagnosis, may carry the potential to change this situation and truly capture the potential of perfusion imaging. Recent advances in the training of recurrent neural networks make it possible to predict and classify time series data with high accuracy. Combining physics-based tissue models and deep learning, using either physics-informed neural networks or universal differential equations, simplifies the training process and increases the interpretability of the resulting models. Due to their versatility, these methods will potentially be useful in bridging the gap between microvascular architecture and perfusion parameters, akin to MR fingerprinting in structural MR imaging. Still, further research is urgently needed before these methods may be used in clinical practice. · Machine learning offers promising methods for processing of perfusion data.. · Recurrent neural networks can classify time series with high accuracy.. · Data augmentation is essentially especially when using small datasets.. · Rotkopf LT, Zhang KS, Tavakoli AA et al. Quantitative Analysis of DCE and DSC-MRI: From Kinetic Modeling toDeep Learning. Fortschr Röntgenstr 2022; 194: 975 - 982.

Deep Learning for Depression Detection from Textual Data

Depression is a prevalent sickness, spreading worldwide with potentially serious implications. Timely recognition of emotional responses plays a pivotal function at present, with the profound expansion of social media and users of the internet. Mental illnesses are highly hazardous, stirring more than three hundred million people. Moreover, that is why research is focused on this subject. With the advancements of machine learning and the availability of sample data relevant to depression, there is the possibility of developing an early depression diagnostic system, which is key to lessening the number of afflicted individuals. This paper proposes a productive model by implementing the Long-Short Term Memory (LSTM) model, consisting of two hidden layers and large bias with Recurrent Neural Network (RNN) with two dense layers, to predict depression from text, which can be beneficial in protecting individuals from mental disorders and suicidal affairs. We train RNN on textual data to identify depression from text, semantics, and written content. The proposed framework achieves 99.0% accuracy, higher than its counterpart, frequency-based deep learning models, whereas the false positive rate is reduced. We also compare the proposed model with other models regarding its mean accuracy. The proposed approach indicates the feasibility of RNN and LSTM by achieving exceptional results for early recognition of depression in the emotions of numerous social media subscribers.

Machine Learning-Based Dynamic Modeling for Process Engineering Applications: A Guideline for Simulation and Prediction from Perceptron to Deep Learning

A misusage of machine learning (ML) strategies is usually observed in the process systems engineering literature. This issue is even more evident when dynamic identification is performed. The root of this problem is the gradient explode and vanishing issue related to the recurrent neural networks training. However, after the advent of deep learning, these issues were mitigated. Furthermore, the problem of data structuration is often overlooked during the machine learning model identification in this field. In this scenario, this work proposes a guideline for identifying ML models for the different applications in process systems engineering, which are usually for simulation or prediction purposes. While using the proposed guideline, the work also identifies a virtual analyzer for a pressure swing adsorption unit. In these types of adsorption separations, it is usual that the measurement of the main properties is not done online. Therefore, the virtual analyzer is another contribution of this manuscript. The overall results demonstrate that even though the test provides good performance during the ML model identification, its quality might degenerate over the application domain if the model application is overlooked.

Periodic synchronization of isolated network elements facilitates simulating and inferring gene regulatory networks including stochastic molecular kinetics

BackgroundThe temporal progression of many fundamental processes in cells and organisms, including homeostasis, differentiation and development, are governed by gene regulatory networks (GRNs). GRNs balance fluctuations in the output of their genes, which trace back to the stochasticity of molecular interactions. Although highly desirable to understand life processes, predicting the temporal progression of gene products within a GRN is challenging when considering stochastic events such as transcription factor–DNA interactions or protein production and degradation.ResultsWe report a method to simulate and infer GRNs including genes and biochemical reactions at molecular detail. In our approach, we consider each network element to be isolated from other elements during small time intervals, after which we synchronize molecule numbers across all network elements. Thereby, the temporal behaviour of network elements is decoupled and can be treated by local stochastic or deterministic solutions. We demonstrate the working principle of this modular approach with a repressive gene cascade comprising four genes. By considering a deterministic time evolution within each time interval for all elements, our method approaches the solution of the system of deterministic differential equations associated with the GRN. By allowing genes to stochastically switch between on and off states or by considering stochastic production of gene outputs, we are able to include increasing levels of stochastic detail and approximate the solution of a Gillespie simulation. Thereby, CaiNet is able to reproduce noise-induced bi-stability and oscillations in dynamically complex GRNs. Notably, our modular approach further allows for a simple consideration of deterministic delays. We further infer relevant regulatory connections and steady-state parameters of a GRN of up to ten genes from steady-state measurements by identifying each gene of the network with a single perceptron in an artificial neuronal network and using a gradient decent method originally designed to train recurrent neural networks. To facilitate setting up GRNs and using our simulation and inference method, we provide a fast computer-aided interactive network simulation environment, CaiNet.ConclusionWe developed a method to simulate GRNs at molecular detail and to infer the topology and steady-state parameters of GRNs. Our method and associated user-friendly framework CaiNet should prove helpful to analyze or predict the temporal progression of reaction networks or GRNs in cellular and organismic biology. CaiNet is freely available at https://gitlab.com/GebhardtLab/CaiNet.

Analysis on Artificial Recurrent Neural Network for Sequence Classification of Audio Data

Recurrent neural networks (RNNs) are capable of learning features and long-term dependencies from sequential and time-series data. The RNNs have a stack of non-linear units where at least one connection between units forms a directed cycle. A well-trained RNN can model any dynamical system; however, training RNNs is mostly plagued by issues in learning long-term dependencies. In this paper, we present a survey on RNNs and several new advances for newcomers and professionals in the field. In this paper literature survey on Artificial Recurrent Neural Network has been presented.

Task-Aware Verifiable RNN-Based Policies for Partially Observable Markov Decision Processes

Partially observable Markov decision processes (POMDPs) are models for sequential decision-making under uncertainty and incomplete information. Machine learning methods typically train recurrent neural networks (RNN) as effective representations of POMDP policies that can efficiently process sequential data. However, it is hard to verify whether the POMDP driven by such RNN-based policies satisfies safety constraints, for instance, given by temporal logic specifications. We propose a novel method that combines techniques from machine learning with the field of formal methods: training an RNN-based policy and then automatically extracting a so-called finite-state controller (FSC) from the RNN. Such FSCs offer a convenient way to verify temporal logic constraints. Implemented on a POMDP, they induce a Markov chain, and probabilistic verification methods can efficiently check whether this induced Markov chain satisfies a temporal logic specification. Using such methods, if the Markov chain does not satisfy the specification, a byproduct of verification is diagnostic information about the states in the POMDP that are critical for the specification. The method exploits this diagnostic information to either adjust the complexity of the extracted FSC or improve the policy by performing focused retraining of the RNN. The method synthesizes policies that satisfy temporal logic specifications for POMDPs with up to millions of states, which are three orders of magnitude larger than comparable approaches.

Machine learning astrophysics from 21 cm lightcones: impact of network architectures and signal contamination

ABSTRACT Imaging the cosmic 21 cm signal will map out the first billion years of our Universe. The resulting 3D lightcone (LC) will encode the properties of the unseen first galaxies and physical cosmology. Here, we build on previous work using neural networks (NNs) to infer astrophysical parameters directly from 21 cm LC images. We introduce recurrent neural networks (RNNs), capable of efficiently characterizing the evolution along the redshift axis of 21 cm LC images. Using a large database of simulated cosmic 21 cm LCs, we compare the relative performance in parameter estimation of different network architectures. These including two types of RNNs, which differ in their complexity, as well as a more traditional convolutional neural network (CNN). For the ideal case of no instrumental effects, our simplest and easiest to train RNN performs the best, with a mean squared parameter estimation error (MSE) that is lower by a factor of ≳2 compared with the other architectures studied here, and a factor of ≳8 lower than the previously-studied CNN. We also corrupt the cosmic signal by adding noise expected from a 1000 h integration with the Square Kilometre Array, as well as excising a foreground-contaminated ‘horizon wedge’. Parameter prediction errors increase when the NNs are trained on these contaminated LC images, though recovery is still good even in the most pessimistic case (with R2 ≳ 0.5−0.95). However, we find no notable differences in performance between network architectures on the contaminated images. We argue this is due to the size of our data set, highlighting the need for larger data sets and/or better data augmentation in order to maximize the potential of NNs in 21 cm parameter estimation.

Multitask neural networks for predicting bladder pressure with time series data

Multitask learning (MTL) can improve accuracy over vanilla neural networks in modeling population level time series data. This can be accomplished by assigning the prediction for each individual in the population as a separate task, thereby leveraging the heterogeneity of population level data. Here, we investigate a novel approach by training recurrent neural networks (RNNs) in a multitask setting. We apply this new methodology to experimental data for predicting bladder pressure, and then bladder contractions, from an external urethral sphincter electromyograph (EUS EMG) signal. We found that the multitask models make more accurate individual level predictions than their single tasking counterparts. We observed that, for bladder pressure prediction, either incorporating multitask learning or RNN structure generalized best to out of sample test data and multitasking RNNs had high out of sample correlation coefficients. These results suggest that MTL models could be used to leverage heterogeneous population time series data for making individualized predictions. From these bladder pressure predictions, we predicted the onset of bladder contractions. Our results indicate that the MTL RNN model was superior in both intra- and inter-individual bladder contraction predictions as measured by sensitivity (85.7%), specificity (98.7%) and precision (73.5%).