Dynamic Recurrent Neural Network Research Articles

Research on grid-connected photovoltaic inverter based on quasi-PR controller adjusting by dynamic diagonal recurrent neural network

The single-phase grid-connected photovoltaic inverter system is studied in this paper. In view of the non-linear and time-varying characteristics of this system, the three-closed-loop control strategy consisting of DC voltage outer loop, grid-connected current inner loop and capacitive current inner loop based on quasi-PR control is proposed. Since the quasi-PR controller of fixed parameters is unable to adapt to changes of parameters in power network, a quasi-PR control method of dynamic self-tuning based on a dynamic Diagonal Recurrent Neural Network (DRNN) is presented. DRNN is based on the Recursive Prediction Error (RPE) algorithm with second-order gradient, which has a faster convergence rate than the BP algorithm. The simulation and experiment results prove that the grid-connected photovoltaic inverter with the above control algorithm has a good quality of the output current and fast performance in dynamic response.

Research on grid-connected photovoltaic inverter based on quasi-PR controller adjusting by dynamic diagonal recurrent neural network

Research on grid-connected photovoltaic inverter based on quasi-PR controller adjusting by dynamic diagonal recurrent neural network

A Novel Approach for an MPPT Controller Based on the ADALINE Network Trained with the RTRL Algorithm

The Real-Time Recurrent Learning Gradient (RTRL) algorithm is characterized by being an online learning method for training dynamic recurrent neural networks, which makes it ideal for working with non-linear control systems. For this reason, this paper presents the design of a novel Maximum Power Point Tracking (MPPT) controller with an artificial neural network type Adaptive Linear Neuron (ADALINE), with Finite Impulse Response (FIR) architecture, trained with the RTRL algorithm. With this same network architecture, the Least Mean Square (LMS) algorithm was developed to evaluate the results obtained with the RTRL controller and then make comparisons with the Perturb and Observe (P&O) algorithm. This control method receives as input signals the current and voltage of a photovoltaic module under sudden changes in operating conditions. Additionally, the efficiency of the controllers was appraised with a fuzzy controller and a Nonlinear Autoregressive Network with Exogenous Inputs (NARX) controller, which were developed in previous investigations. It was concluded that the RTRL controller with adaptive training has better results, a faster response, and fewer bifurcations due to sudden changes in the input signals, being the ideal control method for systems that require a real-time response.

Recurrent and Dynamic Models for Predicting Streaming Video Quality of Experience

Streaming video services represent a very large fraction of global bandwidth consumption. Due to the exploding demands of mobile video streaming services, coupled with limited bandwidth availability, video streams are often transmitted through unreliable, low-bandwidth networks. This unavoidably leads to two types of major streaming-related impairments: compression artifacts and/or rebuffering events. In streaming video applications, the end-user is a human observer; hence being able to predict the subjective Quality of Experience (QoE) associated with streamed videos could lead to the creation of perceptually optimized resource allocation strategies driving higher quality video streaming services. We propose a variety of recurrent dynamic neural networks that conduct continuous-time subjective QoE prediction. By formulating the problem as one of time-series forecasting, we train a variety of recurrent neural networks and non-linear autoregressive models to predict QoE using several recently developed subjective QoE databases. These models combine multiple, diverse neural network inputs, such as predicted video quality scores, rebuffering measurements, and data related to memory and its effects on human behavioral responses, using them to predict QoE on video streams impaired by both compression artifacts and rebuffering events. Instead of finding a single time-series prediction model, we propose and evaluate ways of aggregating different models into a forecasting ensemble that delivers improved results with reduced forecasting variance. We also deploy appropriate new evaluation metrics for comparing time-series predictions in streaming applications. Our experimental results demonstrate improved prediction performance that approaches human performance. An implementation of this work can be found at https://github.com/christosbampis/NARX_QoE_release.

Neural network based predictive control of personalized heating systems

The aim of a personalized heating system is to provide a desirable microclimate for each individual when heating is needed. In this paper, we present a method based on machine learning algorithms for generation of predictive models for use in control of personalized heating systems. Data was collected from two individual test subjects in an experiment that consisted of 14 sessions per test subject with each session lasting 4 h. A dynamic recurrent nonlinear autoregressive neural network with exogenous inputs (NARX) was used for developing the models for the prediction of personalized heating settings. The models for subjects A and B were tested with the data that was not used in creating the neural network (unseen data) to evaluate the accuracy of the prediction. Trained NARX showed good performance when tested with the unseen data, with no sign of overfitting. For model A, the optimal network was with 12 hidden neurons with root mean square error equal to 0.043 and Pearson correlation coefficient equal to 0.994. The best result for model B was obtained with a neural network with 16 hidden neurons with root mean square error equal to 0.049 and Pearson correlation coefficient equal to 0.966. In addition to the neural network models, several other machine learning algorithms were tested. Furthermore, the models were on-line tested and the results showed that the test subjects were satisfied with the heating settings that were automatically controlled using the models. Tests with automatic control showed that both test subjects felt comfortable throughout the tests and test subjects expressed their satisfaction with the automatic control.

Nonlinear auto-regressive neural network for mathematical modelling of an airship using experimental data

Autonomous flight of an aerial vehicle requires a sufficiently accurate mathematical model, which can capture system dynamics in the presence of external disturbances. Artificial neural network is known for ideal in capturing systems behaviour, where little knowledge about vehicle dynamics is available. In this paper, we explored this potential of artificial neural network for characterizing nonlinear dynamics of an unmanned airship. The flight experimentation data for an outdoor experimental airship are acquired through a series of pre-determined flight tests. The experimental data are subjected to a class of dynamic recurrent neural network model dubbed as nonlinear auto-regressive model with exogenous inputs for training. Sufficiently trained neural network model captured and demonstrated the longitudinal dynamics of the airship satisfactorily. We also demonstrated the usefulness of proposed technique for Lotte airship, wherein the performance of proposed model is validated and analysed for the Lotte airship flight test data.

Fitting of dynamic recurrent neural network models to sensory stimulus-response data.

We present a theoretical study aiming at model fitting for sensory neurons. Conventional neural network training approaches are not applicable to this problem due to lack of continuous data. Although the stimulus can be considered as a smooth time-dependent variable, the associated response will be a set of neural spike timings (roughly the instants of successive action potential peaks) that have no amplitude information. A recurrent neural network model can be fitted to such a stimulus-response data pair by using the maximum likelihood estimation method where the likelihood function is derived from Poisson statistics of neural spiking. The universal approximation feature of the recurrent dynamical neuron network models allows us to describe excitatory-inhibitory characteristics of an actual sensory neural network with any desired number of neurons. The stimulus data are generated by a phased cosine Fourier series having a fixed amplitude and frequency but a randomly shot phase. Various values of amplitude, stimulus component size, and sample size are applied in order to examine the effect of the stimulus to the identification process. Results are presented in tabular and graphical forms at the end of this text. In addition, to demonstrate the success of this research, a study involving the same model, nominal parameters and stimulus structure, and another study that works on different models are compared to that of this research.

Modeling response of runoff and evapotranspiration for predicting water table depth in arid region using dynamic recurrent neural network

The investigation of the dynamic response of aquifer to rainfall with variation of the atmospheric conditions is a key issue for groundwater resource management. A data-driven dynamic recurrent neural network approach, based on multi-objective optimization is used here to forecast groundwater levels as a function of rainfall, temperature, humidity, runoff and evapotranspiration data. Recurrent Neural Network (RNN) with variable transfer functions like tangential sigmoidal, purelin and logarithmic sigmoidal are used to compute the relative performance of the model. The results demonstrate the transition behavior due to changing evapotranspiration and runoff. Statistical and graphical indicators are used to compare the results. The statistical indicators used in this work are Mean Square Error (MSE), Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Coefficient of Determination (R2). From the results, evapotranspiration loss and run off are the influencing parameters which affect the depth to water table in the ground water reservoir. It is observed that calculated losses due to evapotranspiration are comparatively less during high precipitation in the Rengali province. Results of R2 suggest that inclusion of evapotranspiration and runoff in different scenario improves the model efficiency in predicting water table depth.

Convolutional Neural Networks Applied on Weather Radar Echo Extrapolation

Extrapolation technique of weather radar echo possesses a widely application prospects in short-term nowcast. The traditional methods of radar echo extrapolation are difficult to obtain long limitation period and lacking in utilization rate of radar data. To solve this problem, this paper proposes a method of weather radar echo extrapolation based on convolutional neural networks (CNNs). In order to adapt the strong correlation between weather radar echo images of contiguous time, on the basis of traditional convolutional neural networks, this method present a new CNNs model, namely, Recurrent Dynamic Convolutional Neural Networks (RDCNN). RDCNN consists of recurrent dynamic sub-network and probability prediction layer, and constructs a cyclic structure in the convolution layer, which improve the ability of RDCNN to process time-related images. In the experiments of radar data from Nanjing, Hangzhuo and Xiamen, compared with traditional methods, this method has achieved higher accurate of extrapolation and extended the limitation period effectively, which meets the requirement for application.

LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks.

Recurrent neural networks, and in particular long short-term memory (LSTM) networks, are a remarkably effective tool for sequence modeling that learn a dense black-box hidden representation of their sequential input. Researchers interested in better understanding these models have studied the changes in hidden state representations over time and noticed some interpretable patterns but also significant noise. In this work, we present LSTMVis, a visual analysis tool for recurrent neural networks with a focus on understanding these hidden state dynamics. The tool allows users to select a hypothesis input range to focus on local state changes, to match these states changes to similar patterns in a large data set, and to align these results with structural annotations from their domain. We show several use cases of the tool for analyzing specific hidden state properties on dataset containing nesting, phrase structure, and chord progressions, and demonstrate how the tool can be used to isolate patterns for further statistical analysis. We characterize the domain, the different stakeholders, and their goals and tasks. Long-term usage data after putting the tool online revealed great interest in the machine learning community.

Breeding novel solutions in the brain: a model of Darwinian neurodynamics.

Background: The fact that surplus connections and neurons are pruned during development is well established. We complement this selectionist picture by a proof-of-principle model of evolutionary search in the brain, that accounts for new variations in theory space. We present a model for Darwinian evolutionary search for candidate solutions in the brain. Methods: We combine known components of the brain - recurrent neural networks (acting as attractors), the action selection loop and implicit working memory - to provide the appropriate Darwinian architecture. We employ a population of attractor networks with palimpsest memory. The action selection loop is employed with winners-share-all dynamics to select for candidate solutions that are transiently stored in implicit working memory. Results: We document two processes: selection of stored solutions and evolutionary search for novel solutions. During the replication of candidate solutions attractor networks occasionally produce recombinant patterns, increasing variation on which selection can act. Combinatorial search acts on multiplying units (activity patterns) with hereditary variation and novel variants appear due to (i)noisy recall of patterns from the attractor networks, (ii)noise during transmission of candidate solutions as messages between networks, and, (iii)spontaneously generated, untrained patterns in spurious attractors. Conclusions: Attractor dynamics of recurrent neural networks can be used to model Darwinian search. The proposed architecture can be used for fast search among stored solutions (by selection) and for evolutionary search when novel candidate solutions are generated in successive iterations. Since all the suggested components are present in advanced nervous systems, we hypothesize that the brain could implement a truly evolutionary combinatorial search system, capable of generating novel variants.

Breeding novel solutions in the brain: a model of Darwinian neurodynamics

Background: The fact that surplus connections and neurons are pruned during development is well established. We complement this selectionist picture by a proof-of-principle model of evolutionary search in the brain, that accounts for new variations in theory space. We present a model for Darwinian evolutionary search for candidate solutions in the brain. Methods: We combine known components of the brain – recurrent neural networks (acting as attractors), the action selection loop and implicit working memory – to provide the appropriate Darwinian architecture. We employ a population of attractor networks with palimpsest memory. The action selection loop is employed with winners-share-all dynamics to select for candidate solutions that are transiently stored in implicit working memory. Results: We document two processes: selection of stored solutions and evolutionary search for novel solutions. During the replication of candidate solutions attractor networks occasionally produce recombinant patterns, increasing variation on which selection can act. Combinatorial search acts on multiplying units (activity patterns) with hereditary variation and novel variants appear due to (i) noisy recall of patterns from the attractor networks, (ii) noise during transmission of candidate solutions as messages between networks, and, (iii) spontaneously generated, untrained patterns in spurious attractors. Conclusions: Attractor dynamics of recurrent neural networks can be used to model Darwinian search. The proposed architecture can be used for fast search among stored solutions (by selection) and for evolutionary search when novel candidate solutions are generated in successive iterations. Since all the suggested components are present in advanced nervous systems, we hypothesize that the brain could implement a truly evolutionary combinatorial search system, capable of generating novel variants.

The relation between the continuous and the discrete: A note on the first principles of speech dynamics

The goal of this paper is to show how dynamical theories of phonetics and phonology bridge the dualistic gap between discrete phonological descriptions and continuous phonetic descriptions. By delving into the first principles of dynamics, it is shown that dynamical theories do not assume separate sets of principles to describe discrete and continuous aspects of a system. Rather, the discrete description is shown to predict the continuous one, using the concept of a differential equation, which is thoroughly explained. Linear and nonlinear differential equations are introduced using a discrete approximation, and then used to show how phonological contrast has been accounted for using dynamical systems analysis. A dynamical recurrent neural network model of word formation is then discussed to show how linguistic plans for words are serialized and coordinated into motoric word plans for different articulatory systems in the vocal tract. Furthermore, it is shown that many aspects of the discrete, time-invariant phonological description can be predicted from observed variable continuous phonetic functions, using the principle of least squares and recurrent neural networks.

Wastewater treatment control method based on a rule adaptive recurrent fuzzy neural network

PurposeThe purpose of this paper is to present an on-line modeling and controlling scheme based on the dynamic recurrent neural network for wastewater treatment system.Design/methodology/approachA control strategy based on rule adaptive recurrent neural network (RARFNN) is proposed in this paper to control the dissolved oxygen (DO) concentration and nitrate nitrogen (SNo) concentration. The structure of the RARFNN is self-organized by a rule adaptive algorithm, and the rule adaptive algorithm considers the overall information processing ability of neural network. Furthermore, a stability analysis method is given to prove the convergence of the proposed RARFNN.FindingsBy application in the control problem of wastewater treatment process (WWTP), results show that the proposed control method achieves better performance compared to other methods.Originality/valueThe proposed on-line modeling and controlling method uses the RARFNN to model and control the dynamic WWTP. The RARFNN can adjust its structure and parameters according to the changes of biochemical reactions and pollutant concentrations. And, the rule adaptive mechanism considers the overall information processing ability judgment of the neural network, which can ensure that the neural network contains the information of the biochemical reactions.

Integrated built-in-test false and missed alarms reduction based on forward infinite impulse response & recurrent finite impulse response dynamic neural networks

Built-in tests (BITs) are widely used in mechanical systems to perform state identification, whereas the BIT false and missed alarms cause trouble to the operators or beneficiaries to make correct judgments. Artificial neural networks (ANN) are previously used for false and missed alarms identification, which has the features such as self-organizing and self-study. However, these ANN models generally do not incorporate the temporal effect of the bottom-level threshold comparison outputs and the historical temporal features are not fully considered. To improve the situation, this paper proposes a new integrated BIT design methodology by incorporating a novel type of dynamic neural networks (DNN) model. The new DNN model is termed as Forward IIR & Recurrent FIR DNN (FIRF-DNN), where its component neurons, network structures, and input/output relationships are discussed. The condition monitoring false and missed alarms reduction implementation scheme based on FIRF-DNN model is also illustrated, which is composed of three stages including model training, false and missed alarms detection, and false and missed alarms suppression. Finally, the proposed methodology is demonstrated in the application study and the experimental results are analyzed.

Multiplex visibility graphs to investigate recurrent neural network dynamics

A recurrent neural network (RNN) is a universal approximator of dynamical systems, whose performance often depends on sensitive hyperparameters. Tuning them properly may be difficult and, typically, based on a trial-and-error approach. In this work, we adopt a graph-based framework to interpret and characterize internal dynamics of a class of RNNs called echo state networks (ESNs). We design principled unsupervised methods to derive hyperparameters configurations yielding maximal ESN performance, expressed in terms of prediction error and memory capacity. In particular, we propose to model time series generated by each neuron activations with a horizontal visibility graph, whose topological properties have been shown to be related to the underlying system dynamics. Successively, horizontal visibility graphs associated with all neurons become layers of a larger structure called a multiplex. We show that topological properties of such a multiplex reflect important features of ESN dynamics that can be used to guide the tuning of its hyperparamers. Results obtained on several benchmarks and a real-world dataset of telephone call data records show the effectiveness of the proposed methods.

Dynamic Recurrent Neural Network Based Indirect Adaptive Control for Nonlinear Systems

In this study, an indirect adaptive controller based on dynamic recurrent neural network (DRNN) is developed in the form of the internal model control structure. This control method includes two learning phases, i.e., off-line and on-line learning. In the offline learning, the DRNN is learned by the epoch wise back propagation through time (BPTT) method to represent the forward dynamics of the system to be controlled. In the online phase, the DRNN is used as the internal model of the controlled system and its parameters can be trained by the Truncated BPTT to cope with the possible change in the system dynamics. Hence, the mathematical inversion of the DRNN internal model is computed online to act as the forward controller. Finally, the controller is then obtained by cascading this inverse model with a robust filter and a linear compensator to improve the closed loop performance. The proposed method is applied to a continuous stirred tank reactor (CSTR) with time varying behavior to evaluate the performance of controller.

Spatial-Temporal Separation Based on the Dynamic Recurrent Wavelet Neural Network Modelling for ASP Flooding

In this paper, a three-dimensional spatial-temporal decomposition modelling method is proposed to build the alkali-surfactant-polymer (ASP) flooding model, in which a new dynamic recurrent wavelet neural network (DRWNN) is presented to identify the temporal coefficients. At first, the detailed mathematical model of ASP flooding is described which is a complex distributed parameter system. Then a three-dimensional spatial-temporal modelling method is inferred based on Karhunen-Loeve (K-L) decomposition to decompose the water saturation of reservoir into a series of spatial basis functions and corresponding temporal coefficients. Furthermore, the recurrent wavelet neural network is used to acquire the identification model, in which the injection concentrations of ASP flooding and temporal coefficients are taken as the input and output information. In order to improve the capability of dynamic modelling, DRWNN is proposed through adding feedback layers and setting the different weights with time to achieve dynamic memory of the past information. Considering the gradient descent method for the neural networks training easily leads to local minimum and slow convergence speed, the spectral conjugate gradient method is introduced to optimize the weights of DRWNN. At last, DRWNN is used to build the relation between the moisture content of production wells and the water saturation of the corresponding grids. Thus, the final approximate model of ASP flooding is finished. The accuracy is proved by model with four injection wells and nine production wells through data from the mechanism model.

An unmanned marine vehicle thruster fault diagnosis scheme based on OFNDA

ABSTRACTIn recent years, there has been a growing interest in the use of fault analysis techniques in unmanned marine vehicles (UMVs) owing to their significant impact on marine operations. This study presents a novel approach to the diagnosis of unbalanced load (blades damage) faults in an electric thruster motor in UMV propulsion systems based on orthogonal fuzzy neighbourhood discriminative analysis for feature dimensionality reduction. The diagnosis approach is based on the use of discrete wavelet transforms as a feature extraction tool and the optimal number of mother wavelet function and levels of resolution by analysing the vibration and current signals. As a result of analysis and comparisons, the Deubechies 12 (db12) wavelet and level 8 were chosen. A dynamic recurrent neural network was chosen for fault classification and level of fault severity prediction was implemented. Four faulty conditions were analysed under laboratory conditions and these were recreated by damaging the blades of a motor. The results obtained from the simulation demonstrate the effectiveness and reliability of the proposed methodology in classifying the different faults with greater speed and accuracy compared to existing methods.

Breeding novel solutions in the brain: a model of Darwinian neurodynamics

Background: The fact that surplus connections and neurons are pruned during development is well established. We complement this selectionist picture by a proof-of-principle model of evolutionary search in the brain, that accounts for new variations in theory space. We present a model for Darwinian evolutionary search for candidate solutions in the brain. Methods: We combine known components of the brain – recurrent neural networks (acting as attractors), the action selection loop and implicit working memory – to provide the appropriate Darwinian architecture. We employ a population of attractor networks with palimpsest memory. The action selection loop is employed with winners-share-all dynamics to select for candidate solutions that are transiently stored in implicit working memory. Results: We document two processes: selection of stored solutions and evolutionary search for novel solutions. During the replication of candidate solutions attractor networks occasionally produce recombinant patterns, increasing variation on which selection can act. Combinatorial search acts on multiplying units (activity patterns) with hereditary variation and novel variants appear due to (i) noisy recall of patterns from the attractor networks, (ii) noise during transmission of candidate solutions as messages between networks, and, (iii) spontaneously generated, untrained patterns in spurious attractors. Conclusions: Attractor dynamics of recurrent neural networks can be used to model Darwinian search. The proposed architecture can be used for fast search among stored solutions (by selection) and for evolutionary search when novel candidate solutions are generated in successive iterations. Since all the suggested components are present in advanced nervous systems, we hypothesize that the brain could implement a truly evolutionary combinatorial search system, capable of generating novel variants.