Recurrent Multilayer Perceptron Research Articles

Hybrid neural network models for forecasting ozone and particulate matter concentrations in the Republic of China

Ozone is a toxic gas with massive distinct chemical components from oxygen. Breathing ozone in the air can cause severe effects on human health, especially people who have asthma. It can cause long-lasting damage to the lungs and heart attacks and might lead to death. Forecasting the ozone concentration levels and related pollutant attribute is critical for developing sophisticated environment safety policies. In this paper, we present three artificial neural network (ANN) models to forecast the daily ozone (O3), coarse particulate matter (PM10), and particulate matter (PM2.5) concentrations in a highly polluted city in the Republic of China. The proposed models are (1) recurrent multilayer perceptron (RMLP), (2) recurrent fuzzy neural network (RFNN), and (3) hybridization of RFNN and grey wolf optimizer (GWO), which are referred to as RMLP-ANN, RFNN, and RFNN-GWO models, respectively. The performance of the proposed models is compared with other conventional models previously reported in the literature. The comparative results showed that the proposed models presented outstanding performance. The RFNN-GWO model revealed superior results in the modeling of O3, PM10, and PM2.5 compared with the RMLP-ANN and RFNN models.

Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces with a Recurrent Neural Network

AbstractA recurrent neural network (RNN) called CrackNet‐R is proposed in the article for fully automated pixel‐level crack detection on three‐dimensional (3D) asphalt pavement surfaces. In the article, a new recurrent unit, gated recurrent multilayer perceptron (GRMLP), is proposed to recursively update the internal memory of CrackNet‐R. Unlike the widely used long short‐term memory (LSTM) and gated recurrent unit (GRU), GRMLP is intended for deeper abstractions on the inputs and hidden states by conducting multilayer nonlinear transforms at gating units. CrackNet‐R implements a two‐phase sequence processing: sequence generation and sequence modeling. Sequence generation is specifically developed in the study to find the best local paths that are most likely to form crack patterns. Sequence modeling predicts timely probabilities of the input sequence being a crack pattern. In terms of sequence modeling, GRMLP slightly outperforms LSTM and GRU by using only one more nonlinear layer at each gate. In addition to sequence processing, an output layer is proposed to produce pixel probabilities based on timely probabilities predicted for sequences. The proposed output layer is critical for pixel‐perfect accuracy, as it accomplishes the transition from sequence‐level learning to pixel‐level learning. Using 3,000 diverse 3D images, the training of CrackNet‐R is completed through optimizing sequence modeling, sequence generation, and the output layer serially. The experiment using 500 testing pavement images shows that CrackNet‐R can achieve high Precision (88.89%), Recall (95.00%), and F‐measure (91.84%) simultaneously. Compared with the original CrackNet, CrackNet‐R is about four times faster and introduces tangible improvements in detection accuracy.

Spectrum occupancy prediction based on functional link artificial neural network (FLANN) in ISM band

Recent advancements in artificial neural networks (ANNs) motivated us to design a simple and faster spectrum prediction model termed the functional link artificial neural network (FLANN). The main objective of this paper is to gather realistic data to obtain utilization statistics for the industrial, scientific and medical band of 2.4–2.5 GHz. To present the occupancy statistics, we conducted measurement in indoors at the Swearingen Engineering Center, University of South Carolina. Further, we introduce different threshold-based spectrum prediction schemes to show the impact of threshold on occupancy, and propose a spectrum prediction algorithm based on FLANN to forecast a future spectrum usage profile from historical occupancy statistics. Spectrum occupancy is estimated and predicted by employing different ANN models including the Feed-forward multilayer perceptron (MLP), Recurrent MLP, Chebyshev FLANN and Trigonometric FLANN. It is observed that the absence of a hidden layer in FLANN makes it more efficient than the MLP model in predicting the occupancy faster and with less complexity. A set of illustrative results are presented to validate the performance of our proposed learning scheme.

Neural networks for karst groundwater management: case of the Lez spring (Southern France)

Karst aquifers provide water resources for a large part of the Mediterranean population and water resources become a strategic problem during summer when the population increases due to tourism. To help managers to optimize the exploitation of karst aquifer waters this paper studies the ability of a neural model to efficiently simulate and forecast water table levels a few months ahead during the dry season. The neural model is a recurrent multilayer perceptron that learns the relations between inputs (mainly rainfall and pumping discharge) and output (water table level). After a training step using several years of data, the model is assessed on a never seen year to be validated. Particular attention is devoted to the dry season (May–September). The model achieved good forecast of the maximal observed drawdown. To investigate the usefulness of the model as an operational tool, various scenarios of future rainfalls and future pumping are proposed and associated forecasts are analysed. Scenarios are the following: zero-rainfall scenario and mean forecast rainfall 1 week ahead; in both cases, the model proposes accurate estimations of the maximal drawdown. In a second step, pumping scenarios are investigated: a mean pumping scenario during the dry season and the maximum pumping scenario taking into account the actual pumps. One more time, it is demonstrated that the maximal drawdown can be efficiently forecast for the dry season. The methodology is generic and can thus be used with profit by managers on other karst aquifers.

Comparison of statistical methods for downscaling daily precipitation

There are several statistical downscaling methods available for generating local-scale meteorological variables from large-scale model outputs. There is still no universal single method, or group of methods, that is clearly superior, particularly for downscaling daily precipitation. This paper compares different statistical methods for downscaling daily precipitation from numerical weather prediction model output. Three different methods are considered: (i) hybrids; (ii) neural networks; and (iii) nearest neighbor-based approaches. These methods are implemented in the Saguenay watershed in northeastern Canada. Suites of standard diagnostic measures are computed to evaluate and inter-compare the performances of the downscaling models. Although results of the downscaling experiment show mixed performances, clear patterns emerge with respect to the reproduction of variation in daily precipitation and skill values. Artificial neural network-logistic regression (ANN-Logst), partial least squares (PLS) regression and recurrent multilayer perceptron (RMLP) models yield greater skill values, and conditional resampling method (SDSM) and K-nearest neighbor (KNN)-based models show the potential to capture the variability in daily precipitation.

Improving long-range hydrological forecasts with extended Kalman filters

There is a continuing effort to advance the skill of long-range hydrological forecasts to support water resources decision making. The present study investigates the potential of an extended Kalman filter approach to perform supervised training of a recurrent multilayer perceptron (RMLP) to forecast up to 12-month-ahead lake water levels and streamflows in Canada. The performance of the RMLP was compared with the conventional multilayer perceptron (MLP) using suites of diagnostic measures. The results of the forecasting experiment showed that the RMLP model was able to provide a robust modelling framework capable of describing complex dynamics of the hydrological processes, thereby yielding more accurate and realistic forecasts than the MLP model. The performance of the method in the present study is very promising; however, further investigation is required to ascertain the versatility of the approach in characterizing different water resources and environmental problems. Citation Muluye, G. Y. (2011) Improving long-range hydrological forecasts with extended Kalman filters. Hydrol. Sci. J. 56(7), 1118–1128.

Identification of a nonlinear black-box model for a self-sensing polymer metal compositeactuator

An ion polymer metal composite (IPMC) is an electro-active polymer that bends inresponse to a small applied electrical field as a result of the mobility of cations in thepolymer network and vice versa. The aim of this paper is the identification of a novelaccurate nonlinear black-box model (NBBM) for IPMC actuators with self-sensing behaviorbased on a recurrent multi-layer perceptron neural network (RMLPNN) and aself-adjustable learning mechanism (SALM).Firstly, an IPMC actuator is investigated. Driving voltage signals are applied to the IPMCin order to identify the IPMC characteristics. Secondly, the advanced NBBM for the IPMCis built with suitable inputs and output to estimate the IPMC tip displacement. Finally,the model parameters are optimized by the collected input/output training data.Modeling results show that the proposed self-sensing methodology based on theoptimized NBBM model can well describe the bending behavior of the IPMCactuator corresponding to its applied power without using any measuring sensor.

Towards a generic neural network model for the prediction of daily streamflow in ungauged boreal plain watersheds

Most of the currently available streamflow neural network models are either recurrent neural networks or feed-forward multi-layer perceptron (FF-MLP) requiring past flow values for lead time prediction. These models cannot be used for modelling ungauged watersheds because past flow values are not available in such cases. This study proposes a FF-MLP algorithm that relies only on low-cost, readily available meteorological data and careful time series manipulation prior to model building, and thus, is suitable for modelling streamflow in ungauged watersheds. The proposed approach was tested on four watersheds (5 to 130 km2) in the Canadian Boreal forest and was found to provide an efficient modelling alternative for daily streamflow predictions. To assess the possibility of successful model transferability from a gauged watershed to a hydrologically similar ungauged watershed, a new remotely sensed hydrologic similarity measure — SWMIR_SI — was proposed and was found to provide a successful indicator of basin similarity. The square of Pearson’s correlation coefficient, r2, was evaluated to exceed 0.71 when SWMIR_SI was regressed to models’ “goodness-of-fit” statistics reflecting the usefulness of the approach.

Removal OF EMG and ECG artifacts from EEG based on real time recurrent learning algorithm

EEG (Electroencephalograph) recording from the scalp has biological artifacts and external artifacts. Biological artifacts, which are generated, can be EMG (Electromyograph) signal, EOG (Electrooculograph) signal or ECG (Electrocardiograph) signal. These artifacts appear as noise in the recorded EEG signal individually or in a combined manner. Usually physicians are misled by these noisy signals and the EEG analysis can go wrong. This paper presents noise cancellation i.e. removal of noise signal which can be either EMG, ECG or a combination of these two artifacts from the corrupted EEG signal and also signal enhancement both using recurrent learning technique. For this purpose, we have implemented the RTRL (Real Time Recurrent Learning) algorithm, which is the most recent and sophisticated real time neural networks algorithm. This algorithm is coded using C language and is executed on the DSP processor TMS320CV5402. The obtained result is verified using MATLAB. Key words: Electroencephalograph (EEG), Recurrent Multilayer Perceptron (RMLP), Real Time Recurrent Learning (RTRL).

Online State–Space Modeling Using Recurrent Multilayer Perceptrons with Unscented Kalman Filter

A nonlinear black-box modeling approach using a state--space recurrent multilayer perceptron (RMLP) is considered in this paper. The unscented Kalman filter (UKF), which was proposed recently and is appropriate for state--space representation, is employed to train the RMLP. The UKF offers a derivative-free computation and an easy implementation, compared to the extended Kalman filter (EKF) widely used for training neural networks. In addition, the UKF has a fast convergence rate and an excellent capability of parameter estimation which are appropriate for online learning. Through modeling experiments of nonlinear systems, the effectiveness of the RMLP trained with the UKF is demonstrated.

Learning in dynamic neural networks using signal flow graphs

The paper presents the universal approach to the determination of the sensitivity functions for dynamic neural networks and its application in learning algorithms of adaptive networks. The method is based on the application of signal flow graph and specially defined graph adjoint to it. The method is equally applied to either feed-forward or recurrent network structures. This paper is mainly concerned with neural network applications of the approach. Different kinds of dynamic neural networks are considered and discussed in the paper: the FIR dynamic multilayer perceptron (MLP), the cascade connection of dynamic MLPs as well as two non-linear recurrent systems: the dynamic recurrent MLP network and ARMA recurrent network. The rule of sensitivity determination has been applied in practical learning of neural networks. Chosen results of numerical experiments concerning the application of this approach to the learning processes of recurrent neural networks are also given and discussed.

Application of the recurrent multilayer perceptron in modeling complex process dynamics

A nonlinear dynamic model is developed for a process system, namely a heat exchanger, using the recurrent multilayer perceptron network as the underlying model structure. The perceptron is a dynamic neural network, which appears effective in the input-output modeling of complex process systems. Dynamic gradient descent learning is used to train the recurrent multilayer perceptron, resulting in an order of magnitude improvement in convergence speed over a static learning algorithm used to train the same network. In developing the empirical process model the effects of actuator, process, and sensor noise on the training and testing sets are investigated. Learning and prediction both appear very effective, despite the presence of training and testing set noise, respectively. The recurrent multilayer perceptron appears to learn the deterministic part of a stochastic training set, and it predicts approximately a moving average response of various testing sets. Extensive model validation studies with signals that are encountered in the operation of the process system modeled, that is steps and ramps, indicate that the empirical model can substantially generalize operational transients, including accurate prediction of instabilities not in the training set. However, the accuracy of the model beyond these operational transients has not been investigated. Furthermore, online learning is necessary during some transients and for tracking slowly varying process dynamics. Neural networks based empirical models in some cases appear to provide a serious alternative to first principles models.

Empirical Model Development and Validation with Dynamic Learning in the Recurrent Multilayer Perceptron

A nonlinear multivariable empirical model is developed for a U-tube steam generator using the recurrent multilayer perceptron network as the underlying model structure. The recurrent multilayer perceptron is a dynamic neural network, very effective in the input-output modeling of complex process systems. A dynamic gradient descent learning algorithm is used to train the recurrent multilayer perceptron, resulting in an order of magnitude improvement in convergence speed over static learning algorithms. In developing the U-tube steam generator empirical model, the effects of actuator, process,and sensor noise on the training and testing sets are investigated. Learning and prediction both appear very effective, despite the presence of training and testing set noise, respectively. The recurrent multilayer perceptron appears to learn the deterministic part of a stochastic training set, and it predicts approximately a moving average response. Extensive model validation studies indicate that the empirical model can substantially generalize (extrapolate), though online learning becomes necessary for tracking transients significantly different than the ones included in the training set and slowly varying U-tube steam generator dynamics. In view of the satisfactory modeling accuracy and the associated short development time, neural networks based empirical models in some cases appear to provide a serious alternative to firstmore » principles models. Caution, however, must be exercised because extensive on-line validation of these models is still warranted.« less

Synthetic approach to optimal filtering

As opposed to the analytic approach used in the modern theory of optimal filtering, a synthetic approach is presented. The signal/sensor data, which are generated by either computer simulation or actual experiments, are synthesized into a filter by training a recurrent multilayer perceptron (RMLP) with at least one hidden layer of fully or partially interconnected neurons and with or without output feedbacks. The RMLP, after adequate training, is a recursive filter optimal for the given structure, with the lagged feedbacks carrying the optimal conditional statistics at each time point. Above all, it converges to the minimum variance filter as the number of hidden neurons increases. We call such an RMLP a neural filter. Simulation results show that the neural filters with only a few hidden neurons consistently outperform the extended Kalman filter and even the iterated extended Kalman filter for the simple nonlinear signal/sensor systems considered.

Nonlinear Identification of Process Dynamics Using Neural Networks

In this paper the nonlinear identification of process dynamics encountered in nuclear power plant components is addressed, in an input-output sense, using artificial neural systems. A hybrid feedforward/feedback neural network, namely, a recurrent multilayer perceptron, is used as the model structure to be identified. The feedforward portion of the network architecture provides its well-known interpolation property, while through recurrency and cross-talk, the local information feedback enables representation of temporal variations in the system nonlinearities. The standard backpropagation learning algorithm is modified, and it is used for the supervised training of the proposed hybrid network. The performance of recurrent multilayer perceptron networks in identifying process dynamics is investigated via the case study of a U-tube steam generator. The response of representative steam generator is predicted using a neural network, and it is compared to the response obtained from a sophisticated computer model based on first principles. The transient responses compare well, although further research is warranted to determine the predictive capabilities of these networks during more severe operational transients and accident scenarios.