Elman Recurrent Network Research Articles

Fading memory as inductive bias in residual recurrent networks

Residual connections have been proposed as an architecture-based inductive bias to mitigate the problem of exploding and vanishing gradients and increased task performance in both feed-forward and recurrent networks (RNNs) when trained with the backpropagation algorithm. Yet, little is known about how residual connections in RNNs influence their dynamics and fading memory properties. Here, we introduce weakly coupled residual recurrent networks (WCRNNs) in which residual connections result in well-defined Lyapunov exponents and allow for studying properties of fading memory. We investigate how the residual connections of WCRNNs influence their performance, network dynamics, and memory properties on a set of benchmark tasks. We show that several distinct forms of residual connections yield effective inductive biases that result in increased network expressivity. In particular, those are residual connections that (i) result in network dynamics at the proximity of the edge of chaos, (ii) allow networks to capitalize on characteristic spectral properties of the data, and (iii) result in heterogeneous memory properties. In addition, we demonstrate how our results can be extended to non-linear residuals and introduce a weakly coupled residual initialization scheme that can be used for Elman RNNs.

A permutation approach to evaluating the performance of a forecasting model of methane content in the atmospheric surface layer of arctic region

The time series forecasting models of climate and ecological systems have grown in number, variety and complexity over the past years. In our work, we proposed to use the ideology of testing statistical hypotheses to evaluate the performance of a predictive time series model. The data for this study were obtained during July–August 2017 from the surface concentration dynamics monitoring of the main greenhouse gases at the Arctic Island of Belyy, Yamalo-Nenets Autonomous Okrug, Russia. Models based on the two types of artificial neural networks - simple recurrent Elman network and nonlinear autoregressive model with exogenous input (NARX), predicted changes in methane concentration. Totally, six models were considered: Elman, NARX and their hybrid model using Daubechies wavelet (db4) and Symlet wavelet (sym4) transformed data for training. The performance of the models was evaluated using 12 ″traditional” indices (correlation and coefficient of determination, Wilmott agreement indices, the mean absolute and mean square errors, etc.) and the proposed permutation approach using the ideology of testing statistical hypotheses. In general, on the data under study, the estimates of the permutational approach and other accuracy indicators and error are the same, but there are some discrepancies. Effect sizes do not always determine the statistical significance of differences. Different artificial neural networks forecasting models are tuned to capture different patterns in the time series of greenhouse gases in the atmosphere. The permutational approach is useful in determining the best model for prediction. So, to predict the variability of methane content, the hybrid model autoregressive neural networks NARX (sym4) is the best performing (means difference = −0.00373 ppm; p-value = 0.01327; correlation coefficient = 0.83320; p-value = 0.00000), and to predict the average concentration of methane content for the entire forecast period, the recurrent neural network Elman (sym4) model is the best (means difference = 0.00241 ppm; p-value = 0.17758; correlation coefficient = 0.50388; p-value = 0.01986).

Artificial neural networks for developing early warning system for banking system: Indian context

This study attempts to develop an early warning system for the Indian banking sector using artificial neural networks (ANNs) by considering important economic variables based on detailed literature review. It takes into account an Elman recurrent neural network and a multilayered feedforward backpropagation network (MLFN). The ANNs are evaluated based on their accuracy and calibration using quadratic probability score (QPS) and global squared bias (GSB) for both within the sample and out of the sample. The scores depict results with Elman recurrent network outperforming the MLFN. The uniqueness of this study lies in using and identifying pertinent macroeconomic variables to anticipate the banking sector fragility for the Indian economy using sequential feature selection algorithms. The ANN models are found to be appropriate and useful for policy planners to foresee the possibility of the occurrence of banking fragility and take proactive corrective measures to minimise and safeguard the economy from adverse implications of banking crisis.

Advantages of Combining Factorization Machine with Elman Neural Network for Volatility Forecasting of Stock Market

With a focus in the financial market, stock market dynamics forecasting has received much attention. Predicting stock market fluctuations is usually challenging due to the nonlinear and nonstationary time series of stock prices. The Elman recurrent network is renowned for its capability of dealing with dynamic information, which has made it a successful application to predicting. We developed a hybrid approach which combined Elman recurrent network with factorization machine (FM) technique, i.e., the FM-Elman neural network, to predict stock market volatility. In this paper, the Standard & Poor’s 500 Composite Stock Price (S&P 500) index, the Dow Jones industrial average (DJIA) index, the Shanghai Stock Exchange Composite (SSEC) index, and the Shenzhen Securities Component Index (SZI) were used to demonstrate the validity of our proposed FM-Elman model in time-series prediction. The results were compared with predictions obtained from the other two models which are basic BP neural network and the Elman neural network. Some experiments showed that the FM-Elman model outperforms others through different accuracy measures. Furthermore, the effects of volatility degree on prediction performance from different stock indexes were investigated. An interesting phenomenon had been found through some numerical experiments on the effects of different user-specified dimensions on the proposed FM-Elman neural network.

An intelligent parametric modeling and identification of a 5 kW ballard PEM fuel cell system based on dynamic recurrent networks with delayed context units

This article presents a dynamic simulation study for the modeling and identification of a 5 kW Proton Exchange Membrane (PEM) fuel cell system using intelligent ANN approach to get rid of the complexity involved in the analytical modeling as it is intricate with the highly non-linear dynamics such as electrochemical, thermodynamic and water-transport mechanisms. The proposed artificial networks for the prediction and identification of a highly non-linear fuel cell system performance are Radial Basis Function Network (RBFN), dynamic Elman Recurrent Network (ERN) and NARX Recurrent Network (NRN) that has delayed context unit. A comparative study is made between the performances of the proposed neural network models for identifying an optimal network structure and configuration based on network performance measures. The optimal NARX network ascertained having an appreciable learning and generalization ability is adopted for the prediction and identification of the fuel cell system for its static and dynamic behavior. The network prediction over the system behavior shows good agreement with the benchmark results acquired from a 5 kW-Ballard fuel cell system. The proposed optimal intelligent parametric model can facilitate in replacing a highly non-linear fuel cell system in the fuel cell related model developments that are adopted in several research sectors particularly in transportation applications.

A modified conjugate gradient-based Elman neural network

Elman recurrent network is a representative model with feedback mechanism. Although gradient descent method has been widely used to train Elman network, it frequently leads to slow convergence. According to optimization theory, conjugate gradient method is an alternative strategy in searching the descent direction during training. In this paper, an efficient conjugate gradient method has been presented to reach the optimal solution in two ways: (1) constructing a more effective conjugate coefficient, (2) determining adaptive learning rates in terms of the generalized Armijo search method. Experiments show that the performance of the new algorithm is superior to traditional algorithms, such as gradient descent method and conjugate gradient method. In particular, the new algorithm has better performance than the evolutionary algorithm. Finally, we prove the weak and strong convergence of the presented algorithm, i.e., the gradient norm of the error function with respect to the weight vectors converges to zero and the weight sequence approaches a fixed optimal point.

Interaction recognition and intervention based on context feature fusion of learning behaviors in interactive learning environments

ABSTRACT Interactive learning environments can generate massive learning behavior data and the support of learning behavior big data can ensure the completeness of data analysis and robustness of relationship verification. In this study, learning behaviors are divided into training set and testing set, BP neural network and recurrent Elman network are integrated. Through the training set, the recursive multi-layer feedback neural network model based on the fusion of context features is constructed. The recognition algorithm is designed. From relative standard deviation (RSD) and prediction accuracy (P.A), the model and algorithm designed in this study are more suitable for the recognition and prediction of learning behavior data. Furthermore, the key topological path and interaction process equation of learning behaviors are designed, three kinds of intervention optimization strategies are constructed to serve the daily teaching processes of three courses. After two semesters of learning behavior feature recognition and relationship analysis, learners’ interest, assessment pass rate, or excellence rate have improved significantly. So the intervention optimization strategies of learning behaviors based on the analysis results of models and algorithms can influence the feature distribution and behavior trend.

Effect of topology changes of neural network in classification of weld defects

Non destructive testing play vital role in weld assessments. Time of Flight Diffraction testing is one of the reliable ultrasonic testing and evaluation method, followed globally. In this paper, TOFD testing based automatic weld classification is proposed. Artificial neural Network based algorithm is proposed and architecture, network topology of the feed forward network, Cascade forward network, Elman recurrent network and Radial basis function network are discussed. The classification of weld defects are also analyzed, based on their performance metric, classification accuracy. It also focuses on model development, neural network paradigms, training and testing of the proposed model for the time scale feature set.

Neural network system for identification of non-destructive testing signals

In this article the technique of using neural network algorithms in non-destructive control systems is considered. The block diagram of the neural network system for identification of the safe state of fixed objects is presented. The topology of the radial basis and the recurrent Elman network for the initialization of neural structures is determined. Methods and criteria for training the neural network in the process of non-destructive testing of the object are presented

A recurrent Elman network in conjunction with an electronic nose for fast prediction of optimum fermentation time of black tea

Tea industries enjoy a significant position in the socio-economic ladder for any demographics, especially in India who is the largest producer as well as consumer of the agro-product. While tea ranks only next to water in the pedigree of globally consumed beverages, the imperative fermentation stage in the processing of tea leaves is conventionally monitored through olfactory perception of tea tasters. Recent advances in the field of machine olfaction have witnessed the advent of electronic nose prototypes, which provide a scientific validation to the organoleptic estimations disseminated by the tasters. However, fermentation is a continuous process requiring constant monitoring whose successful completion relies heavily on identification of distinct aroma peaks emanated at optimum instants. Since the fermentation process is integral to the final quality, it is deemed beneficial if the optimum fermentation period can be predicted at an earlier stage. Such preemptive information can mitigate constant monitoring requirements and momentary concentration lapses. Recognizing the time series nature of the data generated during the fermentation process with an electronic nose prototype, we have implemented a recurrent Elman network to predict the optimum fermentation period for different black tea samples. The results showed that the prescribed network could predict the optimum period with confidence at the halfway of the process. The minimal error between the predicted and the actual fermentation period at the halfway point suggests that the proposed model can well be integrated with an electronic nose dedicated for monitoring the fermentation process.

Forecasting Conditional Variance of S&P100 Returns Using Feedforward and Recurrent Neural Networks

It is shown that time series about financial market variables are highly nonlinearly dependent on time. Fluctuations or volatility of returns on assets is one of them. Portfolio managers, option traders and market makers are all interested in volatility forecasting in order to get higher profits and less risky positions. The nonlinear dependence on time is very complex and parametric approaches, and linear models fail. Therefore as nonparametric tools artificial neural networks (ANNs) are candidates to deal with the volatility and/or return forecasting problems. On the other hand, based on the fact that volatility is time varying and that periods of high volatility tend to cluster, the most popular models in modeling volatility are GARCH type models because they can account excess kurtosis and asymmetric effects of financial time series. A standard GARCH(1,1) model usually indicates high persistence in the conditional variance, which may originate from structural changes. Hence it is natural that artificial neural networks (ANN) will be constructed to capture the nonlinear relationship between past return innovations and conditional variance which may be missed by linear regression models. First a usual feedforward, back propagation network is used. The structure of the return data makes FFANN difficult to converge. To overcome this difficulty a neural network with appropriate recurrent connections in the context of nonlinear ARMA models are used. These are the Jordan neural networks (JNN). Then Elman recurrent networks (ENN) and a mixture of the two (EJNN) are also used. The data set consists of returns of the S&P100 index daily closing prices obtained from the S&P100 website. The results indicate that the selected JNN(1,1,1) model has superior performances compared to the standard GARCH(1,1) model. The contribution of this paper can be seen in determining the appropriate NN that is comparable to the standard GARCH(1,1) model and its application in forecasting conditional variance of stock returns. Moreover, from the econometric perspective, NN models are used as a semi-parametric method that combines flexibility of nonparametric methods and the interpretability of parameters of parametric methods.

A new cast-resin transformer thermal model based on recurrent neural networks

Abstract Thermal modeling in the transient condition is very important for cast-resin dry-type transformers. In the present research, two novel dynamic thermal models have been introduced for the cast-resin dry-type transformer. These models are based on two artificial neural networks: the Elman recurrent networks (ELRN) and the nonlinear autoregressive model process with exogenous input (NARX). Using the experimental data, the introduced neural network thermal models have been trained. By selecting a typical transformer, the trained thermal models are validated using additional experimental results and the traditional thermal models. It is shown that the introduced neural network based thermal models have a good performance in temperature prediction of the winding and the cooling air in the cast-resin dry-type transformer. The introduced thermal models are more accurate for the temperature analysis of this transformer and they will be trained easily. Finally, the trained and validated thermal models are employed to evaluate the life-time and the reliability of a typical cast-resin dry-type transformer.

A new propagation modeling technique for ultra-wideband implant body area networks based on a neural network architecture

We report on the results of our investigations into more computationally efficient methods for propagation modeling in the implant environment through the use of neural networks. Our results are applicable to the important case of ultra-wideband systems operating in the implant environment. A propagation modeling engine based on neural networks can help the system designer avoid time-consuming numerical simulations normally utilized for propagation modeling. The contributions of this paper are as follows. We perform a systematic investigation of two neural network architectures for propagation modeling, namely the multilayer perceptron (MLP) and the recurrent Elman network. We perform a quantitative comparison of the results produced by these networks to the results produced by time-consuming numerical methods, such as the finite difference time domain (FDTD) method. Our results indicate that it is indeed possible to obtain path loss results using neural network methods that approach FDTD results within a 3- to 4-dB margin of error. We further illustrate that it is possible to attain this level of accuracy through the simpler MLP structure, and that the increased complexity and training time associated with the Elman network is unwarranted. We also observe that the MLP network can attain this performance through reasonable network sizes of 20–30 neurons.

Evolutionary Load Forecasting using ArtificialNeural Network

The anticipated effort targets to predict the load by utilizing Artificial Neural Networks (ANN). Short term load forecasting acts an important character for the economic, planning and reliable action of power systems. Consequently, numerous statistical techniques have been conventionally projected for these forecasting, but it have become tiring to build an exact functional model. This tiring task can be decreased by using ANN. ANN is a machine which is devised to model in fashion which the human brain does a particular task. The main intension of the anticipated effort is to build a NN model known as an Elman recurrent network on the flat form of MATLAB to simulate the load prediction. We also evaluate the results attained by weather and a non-weather sensitive model

Weight Optimization in Recurrent Neural Networks with Hybrid Metaheuristic Cuckoo Search Techniques for Data Classification

Recurrent neural network (RNN) has been widely used as a tool in the data classification. This network can be educated with gradient descent back propagation. However, traditional training algorithms have some drawbacks such as slow speed of convergence being not definite to find the global minimum of the error function since gradient descent may get stuck in local minima. As a solution, nature inspired metaheuristic algorithms provide derivative-free solution to optimize complex problems. This paper proposes a new metaheuristic search algorithm called Cuckoo Search (CS) based on Cuckoo bird’s behavior to train Elman recurrent network (ERN) and back propagation Elman recurrent network (BPERN) in achieving fast convergence rate and to avoid local minima problem. The proposed CSERN and CSBPERN algorithms are compared with artificial bee colony using BP algorithm and other hybrid variants algorithms. Specifically, some selected benchmark classification problems are used. The simulation results show that the computational efficiency of ERN and BPERN training process is highly enhanced when coupled with the proposed hybrid method.

PM<sub>10</sub> Forecasting Using Soft Computing Techniques

Air quality forecasting has acquired great significance in environmental sciences due to its adverse affects on humans and the environment. The artificial neural network is one of the most common soft computing techniques that can be applied for modeling such complex problem. This study designed air quality forecasting model using three-layer FFNN's and recurrent Elman network to forecast PM10 air pollutant concentrations 1 day advance in Yilan County, Taiwan. Then, the optimal model is selected based on testing performance measurements (RMSE, MAE, r, IA and VAF) and learning time. This study used an hourly historical data set from 1/1/2009-31/12/2011 collected by Dongshan station. The data was entirely pre-processed and cleared form missing and outlier values then transformed into daily average values. The final results showed that the three-layer FFNN with One Step Secant (OSS) training algorithm achieved better results than Elman network with Gradient Descent adaptive learning rate (GDX) training algorithm. Where, the FFNN required the less training time and achieved better performance in forecasting PM10 concentrations. Also, the testing performance measurements shown that the selected daily average input variables in previous day (PM_2.5), relative humidity, PM10, temperature, wind direction and speed is critical to give better forecasting accuracy. Whereas, the testing measurements RMSE = 6.23 μg/m₃, MAE = 4.75 μg/m₃, r = 0.943, IA = 0.964 and VAF = 88.80 in PM₁₀ FFNN forecasting model that used OSS training algorithm.

Neural Optimization of Axial- and Radial-Flow Microreactors for Hydrogen Production by Steam Reforming of Methanol

The reforming process between methanol and steam is an important source of hydrogen. Since conventional tubular reactors require storage and transport facilities, which are risk-prone, microreactors are being investigated as convenient portable sources for in situ generation of hydrogen. A recent study has shown that radial-flow microreactors are more efficient than conventional axial-flow designs. These comparisons were made through mathematical models. Since equation-oriented models are of limited validity under realistic conditions, a neural network approach has been used here to model the process and maximize hydrogen production. By optimizing through a recurrent Elman network, the performance of the radial-flow reactor is enhanced by 20%–30%; larger improvements were seen for the axial-flow geometry, but the overall production of hydrogen was still lower than with radial flow. Based on these results and previous work, further improvements may still be possible through hybrid neural models.

Comparison of MLP and REN classifiers for detection of hemodynamic stress using photoplethysmograph

The aim of this study is to show a comparison of the Multi-Layered Perceptron (MLP) neural network and Recurrent Elman Network (REN) in determining false positives for Pulse Photoplethysmogram (PPG) recorded during rest and recovery phase after exercise. Several time domain features, depicting the signal morphology and time indices were identified for classification and the robustness of the neural networks were examined using Classification Accuracy (CA) and Receiver Operating Characteristics (ROC). The obtained CA’s were 91% (MLP) and 96.75% (REN) for Reflection Index (RI) and 90% (MLP) and 94% (REN) for Stiffness Index (SI). Besides, the REN is slightly better with a ROC index of 0.94485 and 0.96742 for RI and SI. From the obtained results, it can be concluded that the REN was found to detect lower false positives when compared to the MLP network.

Estimation of pressuremeter modulus and limit pressure of clayey soils by various artificial neural network models

The main purpose of the present study is to develop some artificial neural network (ANN) models for the prediction of limit pressure (PL) and pressuremeter modulus (EM) for clayey soils. Moisture content, plasticity index, and SPT values are used as inputs in the ANN models. To get plausible results, the number of hidden layer neurons in all models is varied between 1 and 5. In addition, both linear and nonlinear activation functions are considered for the neurons in output layers while a nonlinear activation function is employed for the neurons in the hidden layers of all models. Logistic activation function is used as a nonlinear activation function. During the modeling studies, total eight different ANN models are constructed. The ANN models having two outputs produced the worst results, independent from activation function. However, for PL, the best results are obtained from the feed-forward neural network with five neurons in the hidden layer, and logistic activation function is employed in the output neuron. For EM, the best model producing the most acceptable results is Elman recurrent network model, which has 4 neurons in the neurons in the hidden layer, and linear activation function is used for the output neuron. Finally, the results show that the ANN models produce the more accurate results than the regression-based models. In the literature, when empirical equations based on regression analysis were used, the best root mean square error (RMSE) values obtained to date for PL and EM have been 0.43 and 5.65, respectively. In this study, RMSE values for PL and EM were found to be 0.20 and 2.99, respectively, by using ANN models. It was observed that using ANN approach drastically increases the prediction accuracy in terms of RMSE criterion.

Role of Hidden Neurons in an Elman Recurrent Neural Network in Classification of Cavitation Signals

This paper is intended to present the outcome of a study conducted on the cavitation data collected from accelerometer which is installed at the down stream of the cavitation test loop, to illustrate that the hidden neurons in an ANN modelling tool, indeed, do have roles to play in percentage of classification of cavitation signal. It sheds light on the role of the hidden neurons in an Elman Recurrent type ANN model which is used to classify the cavitation signals. The results confirmed that the hidden-output connection weights become small as the number of hidden neurons becomes large and also that the trade-off in the learning stability between input-hidden and hidden-output connections exists. The Elman recurrent network propagates data from later processing stage to earlier stage. A copy of the previous values of the hidden units are maintained which allows the network to perform sequence-prediction. In the present work, the optimum number of hidden neurons is evolved through an elaborate trial and error procedure. It is concluded that our approach has a significant improvement in learning and also in classification of cavitation signals.