Class Of Recurrent Neural Networks Research Articles

Detection of Nocturnal Scratching Movements in Patients with Atopic Dermatitis Using Accelerometers and Recurrent Neural Networks

Atopic dermatitis is a chronic inflammatory skin condition affecting both children and adults and is associated with pruritus. A method for objectively quantifying nocturnal scratching events could aid in the development of therapies for atopic dermatitis and other pruritic disorders. High-resolution wrist actigraphy (three-dimensional accelerometer sensors sampled at 20Hz) is a noninvasive method to record movement. This paper presents an algorithm to detect nocturnal scratching events based on actigraphy data. The twofold process consists of segmenting the data into "no motion," "single handed motion," and "both handed motion" followed by discriminating motion segments into scratching and other motion using a bidirectional recurrent neural network classifier. The performance was compared against manually scored infrared video data collected from 24 subjects (6 healthy controls and 18 atopic dermatitis patients) demonstrating an score of 0.68 and a rank correlation of 0.945. The algorithm clearly outperformed a published reference method based on wrist actigraphy ( score of 0.09 and a rank correlation of 0.466). The results suggest that scratching movements can be discriminated from other nocturnal movements accurately.

Online Learning Control of Hydraulic Excavators Based on Echo-State Networks

In some of recent advances in automation of construction equipment, much research has been conducted on the control of hydraulic excavators in both industry and academia for the benefit of safety and efficiency. However, most relevant works have employed model-based control approaches that require a mathematical representation of the target plant. For hydraulic excavators, obtaining a useful dynamic model for control can be challenging due to the nonlinearity of the hydraulic servo system. With this in mind, this paper investigates the feasibility of an online learning control framework based on echo-state networks (ESNs) to the position control of hydraulic excavators. While ESNs are a class of recurrent neural networks, the training of ESNs corresponds to solving a linear regression problem, thus making it suitable for online implementation. By exploiting the dynamic properties of ESNs, the deployed control framework uses the input and output signals of the plant to learn an inverse model, which is then used to simultaneously generate control inputs to track the desired trajectory. Experiments conducted on a 21-ton class hydraulic excavator show the promising results in that accurate tracking is achieved even in the absence of a dynamic model.

OSTSC: Over Sampling for Time Series Classification in R

The OSTSC package is a powerful oversampling approach for classifying univariant, but multinomial time series data in R. This vignette provides a brief overview of the oversampling methodology implemented by the package. A tutorial of the OSTSC package is provided. We begin by providing three test cases for the user to quickly validate the functionality in the package. To demonstrate the performance impact of OSTSC, we then provide two medium size imbalanced time series datasets. Each example applies a TensorFlow implementation of a Long Short-Term Memory (LSTM) classifier - a type of a Recurrent Neural Network (RNN) classifier - to imbalanced time series. The classifier performance is compared with and without oversampling. Finally, larger versions of these two datasets are evaluated to demonstrate the scalability of the package. The examples demonstrate that the OSTSC package improves the performance of RNN classifiers applied to highly imbalanced time series data. In particular, OSTSC is observed to increase the AUC of LSTM from 0.543 to 0.784 on a high frequency trading dataset consisting of 30,000 time series observations.

Hybrid modelling of an off line Arabic handwriting recognition system: results and evaluation

In this paper, we presented a state of the art in the field of Arabic handwriting recognition as well as the techniques used. Then, we detailed the general architecture of an Arabic Handwriting Recognition System (AHRS) and the contributions that we have proposed at each phase: first, an analytical segmentation approach based on a morphological/structural analysis of the entire word and sub word; then, a combination of different features to extract the relevant information from text image; thereafter, a hybrid classification approach using long short term memory recurrent neural network (LSTM-RNN) and different other statistical classifiers to learn many characters shapes simultaneously and to improve the performance of our proposed system; and finally, a post-processing approach to reconstruct the characters, words and lines of the text using Morpho-Syntactic analyser 'AlKhalil Morph Sys'. Our proposed system has shown promising results to solve the difficulties of Arabic handwriting recognition.

Distributed Reservoir Computing with Sparse Readouts [Research Frontier

In a network of agents, a widespread problem is the need to estimate a common underlying function starting from locally distributed measurements. Real-world scenarios may not allow the presence of centralized fusion centers, requiring the development of distributed, message-passing implementations of the standard machine learning training algorithms. In this paper, we are concerned with the distributed training of a particular class of recurrent neural networks, namely echo state networks (ESNs). In the centralized case, ESNs have received considerable attention, due to the fact that they can be trained with standard linear regression routines. Based on this observation, in our previous work we have introduced a decentralized algorithm, framed in the distributed optimization field, in order to train an ESN. In this paper, we focus on an additional sparsity property of the output layer of ESNs, allowing for very efficient implementations of the resulting networks. In order to evaluate the proposed algorithm, we test it on two well-known prediction benchmarks, namely the Mackey-Glass chaotic time series and the 10th order nonlinear auto regressive moving average (NARMA) system.

Persistent Memory in Single Node Delay-Coupled Reservoir Computing

Delays are ubiquitous in biological systems, ranging from genetic regulatory networks and synaptic conductances, to predator/pray population interactions. The evidence is mounting, not only to the presence of delays as physical constraints in signal propagation speed, but also to their functional role in providing dynamical diversity to the systems that comprise them. The latter observation in biological systems inspired the recent development of a computational architecture that harnesses this dynamical diversity, by delay-coupling a single nonlinear element to itself. This architecture is a particular realization of Reservoir Computing, where stimuli are injected into the system in time rather than in space as is the case with classical recurrent neural network realizations. This architecture also exhibits an internal memory which fades in time, an important prerequisite to the functioning of any reservoir computing device. However, fading memory is also a limitation to any computation that requires persistent storage. In order to overcome this limitation, the current work introduces an extended version to the single node Delay-Coupled Reservoir, that is based on trained linear feedback. We show by numerical simulations that adding task-specific linear feedback to the single node Delay-Coupled Reservoir extends the class of solvable tasks to those that require nonfading memory. We demonstrate, through several case studies, the ability of the extended system to carry out complex nonlinear computations that depend on past information, whereas the computational power of the system with fading memory alone quickly deteriorates. Our findings provide the theoretical basis for future physical realizations of a biologically-inspired ultrafast computing device with extended functionality.

Classification of epileptic seizures using wavelet packet log energy and norm entropies with recurrent Elman neural network classifier.

Electroencephalogram shortly termed as EEG is considered as the fundamental segment for the assessment of the neural activities in the brain. In cognitive neuroscience domain, EEG-based assessment method is found to be superior due to its non-invasive ability to detect deep brain structure while exhibiting superior spatial resolutions. Especially for studying the neurodynamic behavior of epileptic seizures, EEG recordings reflect the neuronal activity of the brain and thus provide required clinical diagnostic information for the neurologist. This specific proposed study makes use of wavelet packet based log and norm entropies with a recurrent Elman neural network (REN) for the automated detection of epileptic seizures. Three conditions, normal, pre-ictal and epileptic EEG recordings were considered for the proposed study. An adaptive Weiner filter was initially applied to remove the power line noise of 50Hz from raw EEG recordings. Raw EEGs were segmented into 1s patterns to ensure stationarity of the signal. Then wavelet packet using Haar wavelet with a five level decomposition was introduced and two entropies, log and norm were estimated and were applied to REN classifier to perform binary classification. The non-linear Wilcoxon statistical test was applied to observe the variation in the features under these conditions. The effect of log energy entropy (without wavelets) was also studied. It was found from the simulation results that the wavelet packet log entropy with REN classifier yielded a classification accuracy of 99.70% for normal-pre-ictal, 99.70% for normal-epileptic and 99.85% for pre-ictal-epileptic.

Learning Robot Speech Models to Predict Speech Acts in HRI

Abstract In order to be acceptable and able to “camouflage” into their physio-social context in the long run, robots need to be not just functional, but autonomously psycho-affective as well. This motivates a long term necessity of introducing behavioral autonomy in robots, so they can autonomously communicate with humans without the need of “wizard” intervention. This paper proposes a technique to learn robot speech models from human-robot dialog exchanges. It views the entire exchange in the Automated Planning (AP) paradigm, representing the dialog sequences (speech acts) in the form of action sequences that modify the state of the world upon execution, gradually propelling the state to a desired goal. We then exploit intra-action and inter-action dependencies, encoding them in the form of constraints. We attempt to satisfy these constraints using aweighted maximum satisfiability model known as MAX-SAT, and convert the solution into a speech model. This model could have many uses, such as planning of fresh dialogs. In this study, the learnt model is used to predict speech acts in the dialog sequences using the sequence labeling (predicting future acts based on previously seen ones) capabilities of the LSTM (Long Short Term Memory) class of recurrent neural networks. Encouraging empirical results demonstrate the utility of this learnt model and its long term potential to facilitate autonomous behavioral planning of robots, an aspect to be explored in future works.

Multistability of recurrent neural networks with time-varying delays and nonincreasing activation function

In this paper, we are concerned with a class of recurrent neural networks (RNNs) with nonincreasing activation function. First, based on the fixed point theorem, it is shown that under some conditions, such an n-dimensional neural network with nondecreasing activation function can have at least (4k+3)n equilibrium points. Then, it proves that there is only (4k+3)n equilibria under some conditions, among which (2k+2)n equilibria are locally stable. Besides, by analysis and study of RNNs with nondecreasing activation function, we can also obtain the same number of equilibria for RNNs with nonincreasing activation function. Finally, two simulation examples are given to show effectiveness of the obtained results.

Global robust dissipativity of interval recurrent neural networks with time-varying delay and discontinuous activations.

In this paper, the problems of robust dissipativity and robust exponential dissipativity are discussed for a class of recurrent neural networks with time-varying delay and discontinuous activations. We extend an invariance principle for the study of the dissipativity problem of delay systems to the discontinuous case. Based on the developed theory, some novel criteria for checking the global robust dissipativity and global robust exponential dissipativity of the addressed neural network model are established by constructing appropriate Lyapunov functionals and employing the theory of Filippov systems and matrix inequality techniques. The effectiveness of the theoretical results is shown by two examples with numerical simulations.

Dynamic saliency-driven associative memories based on network potential field

In this paper, we present a framework to construct a general class of recurrent neural networks (RNNs) as associative memories (AMs) for pattern storage and retrieval. Different from the traditional AM models that treat elements of a pattern equally, the proposed framework introduces visual saliency of a target pattern into the AM design process by encoding saliency values into the ellipsoidal basis function (EBF) kernel that calculates the weighted distance between the input and target patterns. Network potential fields (NPFs) are then constructed as the linear combination of EBF/radial basis function (RBF) kernels for auto-associative memories (AAMs) and hetero-associative memories (HAMs), respectively. Sparse and dynamic synapses for both the proposed AAMs and HAMs are determined according to the gradients of the NPFs with high efficiency and without the continuity assumptions of the RNN's activation function, which is usually required by traditional AMs. With the proposed method, the target patterns are assigned as fixed point attractors of the network and the AMs are proven to be able to converge to one of these attractors via Lyapunov analysis. The resulted AAMs and HAMs are demonstrated to have excellent tolerance and robustness to a variety of input noises and corruption via comparative experiments on retrieval and association of 2D color images.

Delay-dependent exponential stability of recurrent neural networks with Markovian jumping parameters and proportional delays

This paper deals with the global exponential stability problem of a class of recurrent neural networks with Markovian jumping parameters and proportional delays. Here the proportional delay is unbounded time-varying, which is different from unbounded distributed delay. The nonlinear transformation \(z(t)=x({\text {e}}^{t})\) transforms the recurrent neural networks with Markovian jumping parameters and proportional delays into the recurrent neural networks with Markovian jumping parameters, constant delays and variable coefficients. By constructing Lyapunov functional, a linear matrix inequality (LMI) approach is developed to establish a new delay-dependent global exponential stability sufficient condition in the mean square, which is related to the size of the proportional delay factor and can be easily checked by utilizing the numerically efficient MATLAB LMI toolbox, and no tuning of parameters is required. Two numerical examples and their simulations are given to illustrate the effectiveness of the obtained results.

Sub-word-based Offline Handwritten Farsi Word Recognition Using Recurrent Neural Network

In this paper, we present a segmentation-based method for offline Farsi handwritten word recognition. Although most segmentation-based systems suffer from segmentation errors within the first stages of recognition, using the inherent features of the Farsi writing script, we have segmented the words into sub-words. Instead of using a single complex classifier with many (N) output classes, we have created N simple recurrent neural network classifiers, each having only true/false outputs with the ability to recognize sub-words. Through the extraction of the number of sub-words in each word, and labeling the position of each sub-word (beginning/middle/end), many of the sub-word classifiers can be pruned, and a few remaining sub-word classifiers can be evaluated during the sub-word recognition stage. The candidate sub-words are then joined together and the closest word from the lexicon is chosen. The proposed method was evaluated using the Iranshahr database, which consists of 17,000 samples of Iranian handwritten city names. The results show the high recognition accuracy of the proposed method.

Multistability analysis of a general class of recurrent neural networks with non-monotonic activation functions and time-varying delays

This paper addresses the multistability for a general class of recurrent neural networks with time-varying delays. Without assuming the linearity or monotonicity of the activation functions, several new sufficient conditions are obtained to ensure the existence of (2K+1)n equilibrium points and the exponential stability of (K+1)n equilibrium points among them for n-neuron neural networks, where K is a positive integer and determined by the type of activation functions and the parameters of neural network jointly. The obtained results generalize and improve the earlier publications. Furthermore, the attraction basins of these exponentially stable equilibrium points are estimated. It is revealed that the attraction basins of these exponentially stable equilibrium points can be larger than their originally partitioned subsets. Finally, three illustrative numerical examples show the effectiveness of theoretical results.

Multistability of Recurrent Neural Networks With Nonmonotonic Activation Functions and Mixed Time Delays

This paper presents new theoretical results on the multistability analysis of a class of recurrent neural networks with nonmonotonic activation functions and mixed time delays. Several sufficient conditions are derived for ascertaining the existence of 3 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> equilibrium points and the exponential stability of 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> equilibrium points via state space partition by using the geometrical properties of activation functions and algebraic properties of nonsingular M-matrix. Compared with existing results, the conditions herein are much more computable with one order less linear matrix inequalities. Furthermore, the attraction basins of these exponentially stable equilibrium points are estimated. It is revealed that the attraction basins of the 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</sup> equilibrium points can be larger than their originally partitioned subspaces. Three numerical examples are elaborated with typical nonmonotonic activation functions to substantiate the efficacy and characteristics of the theoretical results.

Pseudo Almost Automorphic Solutions of Recurrent Neural Networks with Time-Varying Coefficients and Mixed Delays

In this paper, existence, uniqueness and global exponential stability of pseudo almost automorphic solutions for a class of recurrent neural networks with time-varying coefficients and mixed delays are established by employing the fixed point theorem and differential inequality. Numerical example with graphical illustration is given to illuminate our main results.

Global exponential periodicity and stability of recurrent neural networks with multi-proportional delays

In this paper, a class of recurrent neural networks with multi-proportional delays is studied. The nonlinear transformation transforms a class of recurrent neural networks with multi-proportional delays into a class of recurrent neural networks with constant delays and time-varying coefficients. By constructing Lyapunov functional and establishing the delay differential inequality, several delay-dependent and delay-independent sufficient conditions are derived to ensure global exponential periodicity and stability of the system. And several examples and their simulations are given to illustrate the effectiveness of obtained results.

Feature Set Evaluation for Offline Handwriting Recognition Systems: Application to the Recurrent Neural Network Model

The performance of handwriting recognition systems is dependent on the features extracted from the word image. A large body of features exists in the literature, but no method has yet been proposed to identify the most promising of these, other than a straightforward comparison based on the recognition rate. In this paper, we propose a framework for feature set evaluation based on a collaborative setting. We use a weighted vote combination of recurrent neural network (RNN) classifiers, each trained with a particular feature set. This combination is modeled in a probabilistic framework as a mixture model and two methods for weight estimation are described. The main contribution of this paper is to quantify the importance of feature sets through the combination weights, which reflect their strength and complementarity. We chose the RNN classifier because of its state-of-the-art performance. Also, we provide the first feature set benchmark for this classifier. We evaluated several feature sets on the IFN/ENIT and RIMES databases of Arabic and Latin script, respectively. The resulting combination model is competitive with state-of-the-art systems.

Comparative Study of Fuzzy Entropy with Relative Spike Amplitude Features for Recognizing Wake-Sleep Stage 1 EEGs

Electroencephalogram (EEG) based sleep stage analysis considered to be the gold standard method for assessment of sleep architecture. Of importance, transition between the first two stages, wake-sleep stage 1 found to be reliable quantitative tool for drowsiness and fatigue detection. The selection of appropriate feature pattern for EEGs is a quite challenging task due to its non-linear and non-stationary nature of the EEG signals. This research work attempts to provide a comparative study of time influence of time domain feature, relative spike amplitude (RSA) with entropy feature, fuzzy entropy(FE) for recognizing the transition between wake and sleep stage 1. EEGs extracted from offline polysomnography database is used and the extracted RSA and FE wake-sleep stage 1 derived EEG features are further classified using a feedback recurrent Elman neural network (REN) classifier. EEGs are segmented into 1s pattern. Simulation of the REN classifier revealed that the FE feature with REN yields a CA of 99.6% compared to that of with RSA feature.

Stability criteria for recurrent neural networks with time-varying delay based on secondary delay partitioning method.

A secondary delay partitioning method is proposed to study the stability problem for a class of recurrent neural networks (RNNs) with time-varying delay. The total interval of the time-varying delay is first divided into two parts, and then each part is further divided into several subintervals. To deal with the state variables associated with these subintervals, an extended reciprocal convex combination approach and a double integral term with variable upper and lower limits of integral as a Lyapunov functional are proposed, which help to obtain the stability criterion. The main feature of the proposed result is more effective for the RNNs with fast time-varying delay. A numerical example is used to show the effectiveness of the proposed stability result.