Time-delay Recurrent Neural Network Research Articles

Time delay recurrent neural networks via PID control law to complex dynamic systems

This paper presents an extension to the theory on the analysis of stability of complex systems with time delay of fractional order; the previous study is based on the theory of functions of Lyapunov–Krasovskii and the fractional-order PID control law. The analytical results are illustrated by the simulation of fractional nonlinear systems interconnected with time delay which are forced to follow complex trajectories as chaotic systems.

Time-Delay Memristive Recurrent Neural Network and Its Complex Dynamics

This paper focuses on the hybrid effects of memristor characteristics, coupling coefficient and time-delay on the recurrent neural network. We construct a novel time-delay recurrent neural network based on a passive hyperbolic tangent memristor, whose instability condition at equilibrium points is judged according to the Routh criterion. Under the changes of the memristor internal parameter, coupling coefficient and time-delay, some complex dynamic behaviors, including chaos and various coexisting attractors, of the time-delay memristive recurrent neural network are observed in detail through the bifurcation diagram, the largest Lyapunov exponent, and the basin of attraction. Finally, the theoretical results are verified by the circuit simulation through MULTISIM.

Less is more: a new machine-learning methodology for spatiotemporal systems

Machine learning provides a way to use only portions of the variables of a spatiotemporal system to predict its subsequent evolution and consequently avoids the curse of dimensionality. The learning machines employed for this purpose, in essence, are time-delayed recurrent neural networks with multiple input neurons and multiple output neurons. We show in this paper that such kinds of learning machines have a poor generalization ability to variables that have not been trained with. We then present a one-dimensional time-delayed recurrent neural network for the same aim of model-free prediction. It can be trained on different spatial variables in the training stage but initiated by the time series of only one spatial variable, and consequently possess an excellent generalization ability to new variables that have not been trained on. This network presents a new methodology to achieve fine-grained predictions from a learning machine trained on coarse-grained data, and thus provides a new strategy for certain applications such as weather forecasting. Numerical verifications are performed on the Kuramoto coupled oscillators and the Barrio−Varea−Aragon−Maini model.

Accurate and reliable state of charge estimation of lithium ion batteries using time-delayed recurrent neural networks through the identification of overexcited neurons

Various neural network models have been adopted for lithium ion battery state of charge (SOC) estimation with good accuracy. However, problems for battery states estimation from neural networks were usually not reported, which is mainly due to the lack of effective solutions other than a trial and error training process. This paper firstly proposes time-delayed recurrent neural network for lithium ion battery modeling and SOC estimation. Both exceptional performances and unexpected overfitting or poor performances are reported with in-depth analysis of the root cause. With explicit formulation of the network, each hidden neuron’s output is examined. It is discovered that overexcited neurons could be the root cause for unexpected poor performances of the neural network. Without overexcited neurons, expectational SOC estimation accuracy is consistently obtained with estimation error being less than 1% for lithium ion magnesium phosphate (LiFeMgPO4) batteries considering a fair comparison in literature.

Master-Slave Synchronization for Trajectory Tracking Error Using Fractional Order Time-Delay Recurrent Neural Networks via Krasovskii-Lur’e Functional for Chua’s Circuit

This paper presents an application of a Fractional Order Time Delay Neural Networks to chaos synchronization. The two main methodologies, on which the approach is based, are fractional order time-delay recurrent neural networks and the fractional order inverse optimal control for nonlinear systems. The problem of trajectory tracking is studied, based on the fractional order Lyapunov-Krasovskii and Lur’e theory, that achieves the global asymptotic stability of the tracking error between a delayed recurrent neural network and a reference function is obtained. The method is illustrated for the synchronization, the analytic results we present a trajectory tracking simulation of a fractional order time-delay dynamical network and the Fractional Order Chua’s circuits.

Empirical modeling of turbidity removal in a dissolved air flotation system: application of artificial neural networks

Abstract Dissolved air flotation (DAF) is a physical separation process that uses air microbubbles to remove suspended material dispersed in a liquid phase. Even though DAF is considered a well-established unit operation, modeling it is difficult due to the complexity of the phenomena involved, resulting in conceptual models with no practical application. Thereby, the objective of this work was to evaluate empirical modeling efficiency in predicting the turbidity removal dynamic using artificial neural networks applied to a DAF prototype. For the study of the neural network input variables, a two-level, full-factorial design was utilized to verify the statistical significance of the saturation pressure and the saturated water flow rate in relation to the turbidity removal. Using a time-delay recurrent neural network architecture, two empirical models were proposed to simulate the dynamic behavior of the turbidity removal promoted by the DAF prototype. The real-time model provided good predictions with R = 0.9717 and MSE = 1.0482, and the simulation model was also able to predict the process behavior presenting performance criteria equal to R = 0.9475 and MSE = 1.8640.

The forecasting of consumer exchange-traded funds (ETFs) via grey relational analysis (GRA) and artificial neural network (ANN)

Our study uses the grey relational analysis (GRA) and artificial neural network (ANN) models for the prediction of consumer exchange-traded funds (ETFs). We apply eight variables, including the put/call ratio, the EUR/USD exchange rate, the volatility index, the Commodity Research Bureau Index (CRB), the short-term trading index, the New York Stock Exchange Composite Index, inflation, and the interest rate. The GRA model results showed that the NYSE, CRB, EUR/USD, and PCR were the four main variables influencing consumer ETFs. The GRA test results of all the ANN models' data showed that the back propagation neural network (BPN) was the best predictive model. Based on the classification of different percentages of training data, the results of GRA revealed that the radial basis function neural network and the time-delay recurrent neural network exhibited consistent results, compared to BPN and the recurrent neural network. The results also pointed out that different percentages of training data were suitable for predicting consumer ETFs' performance based on high and low grey relationship grade variables. Evidence has shown that the ETFs in Brazil and China are more predictable than those in other countries. All ANN models' results indicated that the use of 10% testing data could predict consumer ETFs better, particularly the ETFs of the United States (US) and those excluding the United States (EX-US). The Diebold–Mariano (DM) test results suggest that the best predictability model for consumer ETFs is BPN, which is significantly superior to other models.

A Soft Ground Reaction Force Sensor System Utilizing Time-Delay Recurrent Neural Network

This paper proposes a novel soft sensor system for gait phase analysis with its real-time ground reaction force sensing methodology utilizing Recurrent Neural Network. As the full ground contact of human feet is important for natural and stable walking, the soft sensor system embedded in a bendable foot module is required. The sensor system includes four soft sensor units placed in a rubber outsole. Each sensor unit was fabricated using elastic material, and it can measure exerted normal force through a simple capacitive sensing method. In order to measure accurate force with the sensor system, it is important to compensate nonlinearity and hysteresis inherited from elastic material properties. Recurrent Neural Network with time delays is adopted to be a suitable solution for compensating these undesirable characteristics due to its capability to handle the dynamic behavior of sequential data. The sensor units were calibrated based on the training results of Time-Delay Recurrent Neural Networks. ${R}^{2}$ of sensor units is over than 0.998 and RMSE has dropped dramatically by 64%. The feasibility of the sensor system was validated throughout a real-time ground force measuring experiment. Four gait phases were successfully analyzed according to the data obtained.

Application of grey relational analysis and artificial neural networks on currency exchange-traded notes (ETNs)

Abstract This study determines which index has the strongest influence on the exchange-trade note (ETN) returns using the grey relational analysis. Results show that the volatility index is the strongest, followed by the S&P 500 stock index, the US dollar index, the CRB index, the Trade index, and the Brent crude oil index. However, the US dollar index has the most significant effect of using the index values of currency ETNs, followed by the S&P 500 stock index, volatility index, Brent crude oil index, the CRB index, and Trade index. This study applies four types of the artificial neural network model, namely, back-propagation neural network (BPN), recurrent neural network (RNN), time-delay recurrent neural network (TDRNN), and radial basis function neural network (RBFNN) to capture the nonlinear tendencies of ETNs for better forecasting accuracy. The paper finds that the RNN and RBFNN models have stronger predictive power among the models, and provides the highest forecasting accuracy for the majority of the currency ETNs. However, the RNN model consistently shows that the low grey relational grades (GRG) variables have the strongest influence on the ETN returns, compared with combining all and high GRG variables. These findings suggest that fund managers and traders can potentially rely on both RNN and RBFNN models, particularly the former, in their applications in financial time-series modeling.

Fractional Order PID Control Law for Trajectory Tracking Using Fractional Order Time-Delay Recurrent Neural Networks for Fractional Order Complex Dynamical Systems

In this paper presents an extension to the theory on the analysisof stability of complex systems with time delay of fractionalorder, the previous study is based on the theory of functionsof Lyapunov-Krasovski and the Fractional Order PID controllaw. The analytical results are illustrated by the simulation offractional nonlinear systems interconnected with time delaywhich are forced to follow complex trajectories as chaoticsystems.Fractional complex dynamical systems, trajectory track-ing, Fractional Order Lyapunov-Krasovskii theory, FractionalOrder PID control law.

Trajectory tracking error using fractional order time-delay recurrent neural networks using Krasovskii-Lur’e functional for Chua’s circuit via inverse optimal control

This paper presents an application of a Fractional Order Time Delay Neural Networks to chaos synchronization. The two main methodologies, on which the approach is based, are fractional order time-delay recurrent neural networks and the fractional order inverse optimal control for nonlinear systems. The problem of trajectory tracking is studied, based on the fractional order Lyapunov-Krasovskii and Lur’e theory, that achieves the global asymptotic stability of the tracking error between a delayed recurrent neural network and a reference function is obtained. The method is illustrated for the synchronization, the analytic results we present a trajectory tracking simulation of a fractional order time-delay dynamical network and the Fractional Order Chua’s circuits

Extended dissipative learning of time-delay recurrent neural networks

The paper addresses the issue of extended dissipative learning for a class of delayed recurrent neural networks. Both time-varying delay and time-invariant delay are taken into account. By choosing appropriate Lyapunov–Krasovkii functionals and utilizing some inequalities, several weight learning rules are developed for ensuring the network to be asymptotically stable and extended dissipative. The existence conditions for these learning strategies consist of a few linear matrix inequalities, which are able to be verified readily by Matlab software. Two numerical examples are employed to show the effectiveness and low conservatism of the proposed learning rules.

Time Delay Recurrent Neural Network for Speech Recognition

In Automatic Speech Recognition(ASR), Time Delay Neural Network (TDNN) has been proven to be an efficient network structure for its strong ability in context modeling. In addition, as a feed-forward neural architecture, it is faster to train TDNN, compared with recurrent neural networks such as Long Short-Term Memory (LSTM). However, different from recurrent neural networks, the context in TDNN is carefully designed and is limited. Although stacking Long Short-Term Memory (LSTM) together with TDNN in order to extend the context information have been proven to be useful, it is too complex and is hard to train. In this paper, we focus on directly extending the context modeling capability of TDNNs by adding recurrent connections. Several new network architectures were investigated. The results on the Switchboard show that the best model significantly outperforms the base line TDNN system and is comparable with TDNN-LSTM architecture. In addition, the training process is much simpler than that of TDNN-LSTM.

Prandtl-Ishlinskii Modeling for Giant Magnetostrictive Actuator Based on Internal Time-Delay Recurrent Neural Network

Micro and nano technology is an indispensable part of modern science and technology. Because of the excellent advantages of high energy density, rapid response and large mechanical force, GMA becomes more promising in precision positioning, microelectronic, and biomedicine field[1], [2]. However, the relationship between input current and output displacement of GMA is hysteresis nonlinearity, which shows that the output of GMA not only relates to the current input value, but also relates to previous output value. Besides, the hysteresis nonlinearity is rate-dependent, so that the output of GMA depends on the input frequency. The intrinsic rate-dependent hysteresis nonlinearity of GMA is the main sticking point preventing its application in the high precision positioning [3], [4]. Therefore, modeling of GMA has long been difficult to study and attracted the attention of researchers. In this paper, the Prandtl-Ishlinskii (PI) model with the parameters self-tuning ability is established by the internal time-delay recurrent neural network (RNN) to describe the hysteresis nonlinearity of GMA. PI model consists of play operator and density function. Play operator is a continuous hysteresis operator, whose output depends on not only the current input but also the previous input. Nevertheless, play operator is rate-independent. Identifying the applicable density function is an important part of modeling PI model of GMA. Neural network has the features of the nonlinear mapping property and high parallel process ability. Therefore it is suitable to be applied to identify the nonlinear model. In this paper, the internal time-delay RNN is used to replace the density function of PI model. The PI model structure identified by the internal time-delay RNN for GMA is shown in Fig. 1. The internal time-delay RNN is set by the input layer, output layer and hidden layer. Where output of play operator $v_{i}(t)$ is the $i$ th input sample of the network at time $t$, $y^{\ast }(t)$ is the output of the network. $^{1}w_{ji}$ and $^{2}w_{j}$ are the weights of input layer and output layer, respectively, $^{h}w_{jk}$ is the weight for the nodes of hidden layer. Compared with feedforward neural network, it has property of memory because there is time delay recurrent existed in the hidden layer. Due to the inherent feedback structure of internal time-delay RNN, it possesses the dynamic characteristics and can adjust the parameters of PI model adaptively. The simulation results of PI model identified by internal time-delay RNN at the different input frequency are shown in Fig. 2. The red solid lines are hysteresis loops measured in the experiments, the blue dotted lines are the hysteresis loops of PI model based on internal time-delay RNN, and the blue solid lines are the modeling error curves. To facilitate simulation, the normalized data is adopted. As shown in Fig.2, the maximum modeling error rate at 1Hz, 10Hz, 50Hz and 100Hz is 0.81%, 1.07%, 1.41% and 2.14%, respectively. The PI model identified in this paper can accurately describe the hysteresis nonlinearity of GMA with the increase of input frequency. Therefore, the ability of precise modeling by PI model based on internal time-delay RNN is certified. The proposed PI model can be used to eliminate the hysteresis nonlinearity at the compensation control of GMA and promote the application of GMA in precision positioning field in the future.

Nonlinear Aeroelastic System Identification Based on Neural Network

This paper focuses on the nonlinear aeroelastic system identification method based on an artificial neural network (ANN) that uses time-delay and feedback elements. A typical two-dimensional wing section with control surface is modelled to illustrate the proposed identification algorithm. The response of the system, which applies a sine-chirp input signal on the control surface, is computed by time-marching-integration. A time-delay recurrent neural network (TDRNN) is employed and trained to predict the pitch angle of the system. The chirp and sine excitation signals are used to verify the identified system. Estimation results of the trained neural network are compared with numerical simulation values. Two types of structural nonlinearity are studied, cubic-spring and friction. The results indicate that the TDRNN can approach the nonlinear aeroelastic system exactly.

Fractional delay segments method on time-delayed recurrent neural networks with impulsive and stochastic effects: An exponential stability approach

Abstract This paper concerns with the problem of exponential stability analysis of time-delayed recurrent neural networks with impulsive and stochastic effects under fractional segments or intervals in delays. The delays in discrete term are assumed to be time-varying and different from existing literature, the discrete delay interval has been separated into fractional segments, which guarantees the availability of lower and upper bounds for feasibility with accuracy. By constructing a suitable Lyapunov–Krasovskii functional (LKF), with the aid of stability theory and inequality techniques, several novel criteria are originated via linear matrix inequalities (LMIs) to ensure the exponential stability of addressed neural networks in the mean square sense. Finally, two numerical examples are presented to substantiate the superiority and effectiveness of our theoretical outcomes.

A NOVEL PROPORTIONAL AND SIMULTANEOUS CONTROL METHOD FOR PROSTHETIC HAND

This paper proposes a novel control method of using the surface electromyogram (sEMG) signals to predict the kinematics of hand and wrist, which will be applied in the prosthetic hand control. Prediction of movement in 3 degree-of-freedoms’ (DoFs’) (wrist flexion/extension (WFE), lateral abduction/adduction (LAA), and hand open/close (HOC)) is investigated in this paper. The proposed control method contains a time-delay recurrent neural network (TDRNN), adopting the previous prediction of the joint angles and the time-delay sEMG signals as the system input. This proposed method uses a batch training based on Levenberg–Marquardt (LM) algorithm to learn the weights of the TDRNN. The trained TDRNN is aimed to achieve simultaneous and proportional regression from human movements of the 3 DoFs to those of the prosthetic hand. Three able-bodied subjects are chosen to participate in the test and demonstrate its feasibility and performance. The offline test result R2 ranges between 0.81 and 0.94. The online test results show that TDRNN reacts faster, which verifies that the method proposed in this paper will be feasible and effective in prosthetic hand control.

Adaptive Exponential Synchronization of Multislave Time-Delayed Recurrent Neural Networks With Lévy Noise and Regime Switching

This paper discusses the problem of adaptive exponential synchronization in mean square for a new neural network model with the following features: 1) the noise is characterized by the Lévy process and the parameters of the model change in line with the Markovian process; 2) the master system is also disturbed by the same Lévy noise; and 3) there are multiple slave systems, and the state matrix of each slave system is an affine function of the state matrices of all slave systems. Based on the Lyapunov functional theory, the generalized Itô's formula, -matrix method, and the adaptive control technique, some criteria are established to ensure the adaptive exponential synchronization in the mean square of the master system and each slave system. Moreover, the update law of the control gain and the dynamic variation of the parameters of the slave systems are provided. Finally, the effectiveness of the synchronization criteria proposed in this paper is verified by a practical example.

The forecast for precious metal indexes and precious metal ETFs: an artificial neural network analysis

This paper applies chaos effect, grey relational analysis (GRA) and artificial neural network (ANN) to forecast the return volatility of precious metals and precious metal ETFs. The results showed that all the data series performed deterministic chaos. The GRA results emphasised in all samples, the West Texas Intermediate (WTI) index represents the greatest influence, followed by stock index, exchange rate, commodity research bureau (CRB) index, volatility index, interest rate and put-call (P/C) ratio. Moreover, the backpropagation network (BPN) model is the most powerful model among four ANN models like BPN, radial basic function (RBP), recurrent neural network (RNN) and time-delay recurrent neural network (TDRNN) in forecasting precious metals and precious metal ETFs. 'All variables' group has stronger influence than high- or low-grey relational grade (GRG) variables for predicting precious metals. Whereas, precious metal ETFs have better forecasting results by using 'all variables' group for iShares Silver Trust (SLV) and ETFs physical platinum shares (PPLT) and high-GRG variables for SPDR gold shares (GLD) and ETFs physical palladium shares (PALL). Therefore, investors and traders can get profit through linkage seven determinants in forecasting precious metals and precious metal ETFs.

Speech Recognition Using MSVQ/TDRNN

본 논문에서는 MSVQ(Multi-Section Vector Quantization)와 시간지연 회귀 신경회로망(TDRNN)을 이용한 하이브리드 구조의 음성인식 방법을 제안한다. MSVQ는 음성의 길이를 일정한 구간 수로 정규화한 코드북을 생성하고, 시간지연 회귀 신경회로망은 이 코드북을 이용하여 음성을 인식한다. 시간지연 회귀 신경회로망은 음성의 시계열 문맥정보를 잘 학습할 수 있는 구조로 구성되었다. 음성특징으로 인지선형예측(PLP) 계수가 사용되었다. 음성인식 실험을 수행한 결과 MSVQ/TDRNN 음성인식기는 97.9 %의 화자독립 음성 인식률을 보였다. This paper presents a method for speech recognition using multi-section vector-quantization (MSVQ) and time-delay recurrent neural network (TDTNN). The MSVQ generates the codebook with normalized uniform sections of voice signal, and the TDRNN performs the speech recognition using the MSVQ codebook. The TDRNN is a time-delay recurrent neural network classifier with two different representations of dynamic context: the time-delayed input nodes represent local dynamic context, while the recursive nodes are able to represent long-term dynamic context of voice signal. The cepstral PLP coefficients were used as speech features. In the speech recognition experiments, the MSVQ/TDRNN speech recognizer shows 97.9 % word recognition rate for speaker independent recognition.