Class Of Recurrent Neural Networks Research Articles

Region stability analysis and tracking control of memristive recurrent neural network

Memristor is firstly postulated by Leon Chua and realized by Hewlett–Packard (HP) laboratory. Research results show that memristor can be used to simulate the synapses of neurons. This paper presents a class of recurrent neural network with HP memristors. Firstly, it shows that memristive recurrent neural network has more compound dynamics than the traditional recurrent neural network by simulations. Then it derives that n dimensional memristive recurrent neural network is composed of 22n2 sub neural networks which do not have a common equilibrium point. By designing the tracking controller, it can make memristive neural network being convergent to the desired sub neural network. At last, two numerical examples are given to verify the validity of our result.

Robustness Analyses and Optimal Sampling Gap of Recurrent Neural Network for Dynamic Matrix Pseudoinversion

This study analyses the robustness and convergence characteristics of a neural network. First, a special class of recurrent neural network (RNN), termed a continuous-time Zhang neural network (CTZNN) model, is presented and investigated for dynamic matrix pseudoinversion. Theoretical analysis of the CTZNN model demonstrates that it has good robustness against various types of noise. In addition, considering the requirements of digital implementation and online computation, the optimal sampling gap for a discrete-time Zhang neural network (DTZNN) model under noisy environments is proposed. Finally, experimental results are presented, which further substantiate the theoretical analyses and demonstrate the effectiveness of the proposed ZNN models for computing a dynamic matrix pseudoinverse under noisy environments.

Word embeddings and recurrent neural networks based on Long-Short Term Memory nodes in supervised biomedical word sense disambiguation

Word sense disambiguation helps identifying the proper sense of ambiguous words in text. With large terminologies such as the UMLS Metathesaurus ambiguities appear and highly effective disambiguation methods are required. Supervised learning algorithm methods are used as one of the approaches to perform disambiguation. Features extracted from the context of an ambiguous word are used to identify the proper sense of such a word. The type of features have an impact on machine learning methods, thus affect disambiguation performance. In this work, we have evaluated several types of features derived from the context of the ambiguous word and we have explored as well more global features derived from MEDLINE using word embeddings. Results show that word embeddings improve the performance of more traditional features and allow as well using recurrent neural network classifiers based on Long-Short Term Memory (LSTM) nodes. The combination of unigrams and word embeddings with an SVM sets a new state of the art performance with a macro accuracy of 95.97 in the MSH WSD data set.

Reconstructing Genetic Regulatory Networks Using Two-Step Algorithms with the Differential Equation Models of Neural Networks

The identification of genetic regulatory networks (GRNs) provides insights into complex cellular processes. A class of recurrent neural networks (RNNs) captures the dynamics of GRN. Algorithms combining the RNN and machine learning schemes were proposed to reconstruct small-scale GRNs using gene expression time series. We present new GRN reconstruction methods with neural networks. The RNN is extended to a class of recurrent multilayer perceptrons (RMLPs) with latent nodes. Our methods contain two steps: the edge rank assignment step and the network construction step. The former assigns ranks to all possible edges by a recursive procedure based on the estimated weights of wires of RNN/RMLP (RERNN/RERMLP), and the latter constructs a network consisting of top-ranked edges under which the optimized RNN simulates the gene expression time series. The particle swarm optimization (PSO) is applied to optimize the parameters of RNNs and RMLPs in a two-step algorithm. The proposed RERNN-RNN and RERMLP-RNN algorithms are tested on synthetic and experimental gene expression time series of small GRNs of about 10 genes. The experimental time series are from the studies of yeast cell cycle regulated genes and E. coli DNA repair genes. The unstable estimation of RNN using experimental time series having limited data points can lead to fairly arbitrary predicted GRNs. Our methods incorporate RNN and RMLP into a two-step structure learning procedure. Results show that the RERMLP using the RMLP with a suitable number of latent nodes to reduce the parameter dimension often result in more accurate edge ranks than the RERNN using the regularized RNN on short simulated time series. Combining by a weighted majority voting rule the networks derived by the RERMLP-RNN using different numbers of latent nodes in step one to infer the GRN, the method performs consistently and outperforms published algorithms for GRN reconstruction on most benchmark time series. The framework of two-step algorithms can potentially incorporate with different nonlinear differential equation models to reconstruct the GRN.

Detection of Nocturnal Scratching Movements in Patients with Atopic Dermatitis Using Accelerometers and Recurrent Neural Networks

Atopic dermatitis is a chronic inflammatory skin condition affecting both children and adults and is associated with pruritus. A method for objectively quantifying nocturnal scratching events could aid in the development of therapies for atopic dermatitis and other pruritic disorders. High-resolution wrist actigraphy (three-dimensional accelerometer sensors sampled at 20Hz) is a noninvasive method to record movement. This paper presents an algorithm to detect nocturnal scratching events based on actigraphy data. The twofold process consists of segmenting the data into "no motion," "single handed motion," and "both handed motion" followed by discriminating motion segments into scratching and other motion using a bidirectional recurrent neural network classifier. The performance was compared against manually scored infrared video data collected from 24 subjects (6 healthy controls and 18 atopic dermatitis patients) demonstrating an score of 0.68 and a rank correlation of 0.945. The algorithm clearly outperformed a published reference method based on wrist actigraphy ( score of 0.09 and a rank correlation of 0.466). The results suggest that scratching movements can be discriminated from other nocturnal movements accurately.

Online Learning Control of Hydraulic Excavators Based on Echo-State Networks

In some of recent advances in automation of construction equipment, much research has been conducted on the control of hydraulic excavators in both industry and academia for the benefit of safety and efficiency. However, most relevant works have employed model-based control approaches that require a mathematical representation of the target plant. For hydraulic excavators, obtaining a useful dynamic model for control can be challenging due to the nonlinearity of the hydraulic servo system. With this in mind, this paper investigates the feasibility of an online learning control framework based on echo-state networks (ESNs) to the position control of hydraulic excavators. While ESNs are a class of recurrent neural networks, the training of ESNs corresponds to solving a linear regression problem, thus making it suitable for online implementation. By exploiting the dynamic properties of ESNs, the deployed control framework uses the input and output signals of the plant to learn an inverse model, which is then used to simultaneously generate control inputs to track the desired trajectory. Experiments conducted on a 21-ton class hydraulic excavator show the promising results in that accurate tracking is achieved even in the absence of a dynamic model.

OSTSC: Over Sampling for Time Series Classification in R

The OSTSC package is a powerful oversampling approach for classifying univariant, but multinomial time series data in R. This vignette provides a brief overview of the oversampling methodology implemented by the package. A tutorial of the OSTSC package is provided. We begin by providing three test cases for the user to quickly validate the functionality in the package. To demonstrate the performance impact of OSTSC, we then provide two medium size imbalanced time series datasets. Each example applies a TensorFlow implementation of a Long Short-Term Memory (LSTM) classifier - a type of a Recurrent Neural Network (RNN) classifier - to imbalanced time series. The classifier performance is compared with and without oversampling. Finally, larger versions of these two datasets are evaluated to demonstrate the scalability of the package. The examples demonstrate that the OSTSC package improves the performance of RNN classifiers applied to highly imbalanced time series data. In particular, OSTSC is observed to increase the AUC of LSTM from 0.543 to 0.784 on a high frequency trading dataset consisting of 30,000 time series observations.

Hybrid modelling of an off line Arabic handwriting recognition system: results and evaluation

In this paper, we presented a state of the art in the field of Arabic handwriting recognition as well as the techniques used. Then, we detailed the general architecture of an Arabic Handwriting Recognition System (AHRS) and the contributions that we have proposed at each phase: first, an analytical segmentation approach based on a morphological/structural analysis of the entire word and sub word; then, a combination of different features to extract the relevant information from text image; thereafter, a hybrid classification approach using long short term memory recurrent neural network (LSTM-RNN) and different other statistical classifiers to learn many characters shapes simultaneously and to improve the performance of our proposed system; and finally, a post-processing approach to reconstruct the characters, words and lines of the text using Morpho-Syntactic analyser 'AlKhalil Morph Sys'. Our proposed system has shown promising results to solve the difficulties of Arabic handwriting recognition.

Distributed Reservoir Computing with Sparse Readouts [Research Frontier

In a network of agents, a widespread problem is the need to estimate a common underlying function starting from locally distributed measurements. Real-world scenarios may not allow the presence of centralized fusion centers, requiring the development of distributed, message-passing implementations of the standard machine learning training algorithms. In this paper, we are concerned with the distributed training of a particular class of recurrent neural networks, namely echo state networks (ESNs). In the centralized case, ESNs have received considerable attention, due to the fact that they can be trained with standard linear regression routines. Based on this observation, in our previous work we have introduced a decentralized algorithm, framed in the distributed optimization field, in order to train an ESN. In this paper, we focus on an additional sparsity property of the output layer of ESNs, allowing for very efficient implementations of the resulting networks. In order to evaluate the proposed algorithm, we test it on two well-known prediction benchmarks, namely the Mackey-Glass chaotic time series and the 10th order nonlinear auto regressive moving average (NARMA) system.

Persistent Memory in Single Node Delay-Coupled Reservoir Computing

Delays are ubiquitous in biological systems, ranging from genetic regulatory networks and synaptic conductances, to predator/pray population interactions. The evidence is mounting, not only to the presence of delays as physical constraints in signal propagation speed, but also to their functional role in providing dynamical diversity to the systems that comprise them. The latter observation in biological systems inspired the recent development of a computational architecture that harnesses this dynamical diversity, by delay-coupling a single nonlinear element to itself. This architecture is a particular realization of Reservoir Computing, where stimuli are injected into the system in time rather than in space as is the case with classical recurrent neural network realizations. This architecture also exhibits an internal memory which fades in time, an important prerequisite to the functioning of any reservoir computing device. However, fading memory is also a limitation to any computation that requires persistent storage. In order to overcome this limitation, the current work introduces an extended version to the single node Delay-Coupled Reservoir, that is based on trained linear feedback. We show by numerical simulations that adding task-specific linear feedback to the single node Delay-Coupled Reservoir extends the class of solvable tasks to those that require nonfading memory. We demonstrate, through several case studies, the ability of the extended system to carry out complex nonlinear computations that depend on past information, whereas the computational power of the system with fading memory alone quickly deteriorates. Our findings provide the theoretical basis for future physical realizations of a biologically-inspired ultrafast computing device with extended functionality.

Classification of epileptic seizures using wavelet packet log energy and norm entropies with recurrent Elman neural network classifier.

Electroencephalogram shortly termed as EEG is considered as the fundamental segment for the assessment of the neural activities in the brain. In cognitive neuroscience domain, EEG-based assessment method is found to be superior due to its non-invasive ability to detect deep brain structure while exhibiting superior spatial resolutions. Especially for studying the neurodynamic behavior of epileptic seizures, EEG recordings reflect the neuronal activity of the brain and thus provide required clinical diagnostic information for the neurologist. This specific proposed study makes use of wavelet packet based log and norm entropies with a recurrent Elman neural network (REN) for the automated detection of epileptic seizures. Three conditions, normal, pre-ictal and epileptic EEG recordings were considered for the proposed study. An adaptive Weiner filter was initially applied to remove the power line noise of 50Hz from raw EEG recordings. Raw EEGs were segmented into 1s patterns to ensure stationarity of the signal. Then wavelet packet using Haar wavelet with a five level decomposition was introduced and two entropies, log and norm were estimated and were applied to REN classifier to perform binary classification. The non-linear Wilcoxon statistical test was applied to observe the variation in the features under these conditions. The effect of log energy entropy (without wavelets) was also studied. It was found from the simulation results that the wavelet packet log entropy with REN classifier yielded a classification accuracy of 99.70% for normal-pre-ictal, 99.70% for normal-epileptic and 99.85% for pre-ictal-epileptic.

Learning Robot Speech Models to Predict Speech Acts in HRI

Abstract In order to be acceptable and able to “camouflage” into their physio-social context in the long run, robots need to be not just functional, but autonomously psycho-affective as well. This motivates a long term necessity of introducing behavioral autonomy in robots, so they can autonomously communicate with humans without the need of “wizard” intervention. This paper proposes a technique to learn robot speech models from human-robot dialog exchanges. It views the entire exchange in the Automated Planning (AP) paradigm, representing the dialog sequences (speech acts) in the form of action sequences that modify the state of the world upon execution, gradually propelling the state to a desired goal. We then exploit intra-action and inter-action dependencies, encoding them in the form of constraints. We attempt to satisfy these constraints using aweighted maximum satisfiability model known as MAX-SAT, and convert the solution into a speech model. This model could have many uses, such as planning of fresh dialogs. In this study, the learnt model is used to predict speech acts in the dialog sequences using the sequence labeling (predicting future acts based on previously seen ones) capabilities of the LSTM (Long Short Term Memory) class of recurrent neural networks. Encouraging empirical results demonstrate the utility of this learnt model and its long term potential to facilitate autonomous behavioral planning of robots, an aspect to be explored in future works.

Multistability of recurrent neural networks with time-varying delays and nonincreasing activation function

In this paper, we are concerned with a class of recurrent neural networks (RNNs) with nonincreasing activation function. First, based on the fixed point theorem, it is shown that under some conditions, such an n-dimensional neural network with nondecreasing activation function can have at least (4k+3)n equilibrium points. Then, it proves that there is only (4k+3)n equilibria under some conditions, among which (2k+2)n equilibria are locally stable. Besides, by analysis and study of RNNs with nondecreasing activation function, we can also obtain the same number of equilibria for RNNs with nonincreasing activation function. Finally, two simulation examples are given to show effectiveness of the obtained results.

Global robust dissipativity of interval recurrent neural networks with time-varying delay and discontinuous activations.

In this paper, the problems of robust dissipativity and robust exponential dissipativity are discussed for a class of recurrent neural networks with time-varying delay and discontinuous activations. We extend an invariance principle for the study of the dissipativity problem of delay systems to the discontinuous case. Based on the developed theory, some novel criteria for checking the global robust dissipativity and global robust exponential dissipativity of the addressed neural network model are established by constructing appropriate Lyapunov functionals and employing the theory of Filippov systems and matrix inequality techniques. The effectiveness of the theoretical results is shown by two examples with numerical simulations.

Dynamic saliency-driven associative memories based on network potential field

In this paper, we present a framework to construct a general class of recurrent neural networks (RNNs) as associative memories (AMs) for pattern storage and retrieval. Different from the traditional AM models that treat elements of a pattern equally, the proposed framework introduces visual saliency of a target pattern into the AM design process by encoding saliency values into the ellipsoidal basis function (EBF) kernel that calculates the weighted distance between the input and target patterns. Network potential fields (NPFs) are then constructed as the linear combination of EBF/radial basis function (RBF) kernels for auto-associative memories (AAMs) and hetero-associative memories (HAMs), respectively. Sparse and dynamic synapses for both the proposed AAMs and HAMs are determined according to the gradients of the NPFs with high efficiency and without the continuity assumptions of the RNN's activation function, which is usually required by traditional AMs. With the proposed method, the target patterns are assigned as fixed point attractors of the network and the AMs are proven to be able to converge to one of these attractors via Lyapunov analysis. The resulted AAMs and HAMs are demonstrated to have excellent tolerance and robustness to a variety of input noises and corruption via comparative experiments on retrieval and association of 2D color images.

Delay-dependent exponential stability of recurrent neural networks with Markovian jumping parameters and proportional delays

This paper deals with the global exponential stability problem of a class of recurrent neural networks with Markovian jumping parameters and proportional delays. Here the proportional delay is unbounded time-varying, which is different from unbounded distributed delay. The nonlinear transformation \(z(t)=x({\text {e}}^{t})\) transforms the recurrent neural networks with Markovian jumping parameters and proportional delays into the recurrent neural networks with Markovian jumping parameters, constant delays and variable coefficients. By constructing Lyapunov functional, a linear matrix inequality (LMI) approach is developed to establish a new delay-dependent global exponential stability sufficient condition in the mean square, which is related to the size of the proportional delay factor and can be easily checked by utilizing the numerically efficient MATLAB LMI toolbox, and no tuning of parameters is required. Two numerical examples and their simulations are given to illustrate the effectiveness of the obtained results.

Sub-word-based Offline Handwritten Farsi Word Recognition Using Recurrent Neural Network

In this paper, we present a segmentation-based method for offline Farsi handwritten word recognition. Although most segmentation-based systems suffer from segmentation errors within the first stages of recognition, using the inherent features of the Farsi writing script, we have segmented the words into sub-words. Instead of using a single complex classifier with many (N) output classes, we have created N simple recurrent neural network classifiers, each having only true/false outputs with the ability to recognize sub-words. Through the extraction of the number of sub-words in each word, and labeling the position of each sub-word (beginning/middle/end), many of the sub-word classifiers can be pruned, and a few remaining sub-word classifiers can be evaluated during the sub-word recognition stage. The candidate sub-words are then joined together and the closest word from the lexicon is chosen. The proposed method was evaluated using the Iranshahr database, which consists of 17,000 samples of Iranian handwritten city names. The results show the high recognition accuracy of the proposed method.

Multistability analysis of a general class of recurrent neural networks with non-monotonic activation functions and time-varying delays

This paper addresses the multistability for a general class of recurrent neural networks with time-varying delays. Without assuming the linearity or monotonicity of the activation functions, several new sufficient conditions are obtained to ensure the existence of (2K+1)n equilibrium points and the exponential stability of (K+1)n equilibrium points among them for n-neuron neural networks, where K is a positive integer and determined by the type of activation functions and the parameters of neural network jointly. The obtained results generalize and improve the earlier publications. Furthermore, the attraction basins of these exponentially stable equilibrium points are estimated. It is revealed that the attraction basins of these exponentially stable equilibrium points can be larger than their originally partitioned subsets. Finally, three illustrative numerical examples show the effectiveness of theoretical results.

Multistability of Recurrent Neural Networks With Nonmonotonic Activation Functions and Mixed Time Delays

This paper presents new theoretical results on the multistability analysis of a class of recurrent neural networks with nonmonotonic activation functions and mixed time delays. Several sufficient conditions are derived for ascertaining the existence of 3 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> equilibrium points and the exponential stability of 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> equilibrium points via state space partition by using the geometrical properties of activation functions and algebraic properties of nonsingular M-matrix. Compared with existing results, the conditions herein are much more computable with one order less linear matrix inequalities. Furthermore, the attraction basins of these exponentially stable equilibrium points are estimated. It is revealed that the attraction basins of the 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> equilibrium points can be larger than their originally partitioned subspaces. Three numerical examples are elaborated with typical nonmonotonic activation functions to substantiate the efficacy and characteristics of the theoretical results.

Pseudo Almost Automorphic Solutions of Recurrent Neural Networks with Time-Varying Coefficients and Mixed Delays

In this paper, existence, uniqueness and global exponential stability of pseudo almost automorphic solutions for a class of recurrent neural networks with time-varying coefficients and mixed delays are established by employing the fixed point theorem and differential inequality. Numerical example with graphical illustration is given to illuminate our main results.