Nonlinear Recurrent Neural Network Research Articles

Multi-task learning using non-linear autoregressive models and recurrent neural networks for tide level forecasting

Tide level forecasting plays an important role in environmental management and development. Current tide level forecasting methods are usually implemented for solving single task problems, that is, a model built based on the tide level data at an individual location is only used to forecast tide level of the same location but is not used for tide forecasting at another location. This study proposes a new method for tide level prediction at multiple locations simultaneously. The method combines nonlinear autoregressive moving average with exogenous inputs (NARMAX) model and recurrent neural networks (RNNs), and incorporates them into a multi-task learning (MTL) framework. Experiments are designed and performed to compare single task learning (STL) and MTL with and without using non-linear autoregressive models. Three different RNN variants, namely, long short-term memory (LSTM), gated recurrent unit (GRU) and bidirectional LSTM (BiLSTM) are employed together with non-linear autoregressive models. A case study on tide level forecasting at many different geographical locations (5 to 11 locations) is conducted. Experimental results demonstrate that the proposed architectures outperform the classical single-task prediction methods.

Wetland Mapping by Fusion of Deep learning and Ensemble Model for Enhancing Prediction Outcomes

Constraints perceived in different socioeconomic situations reinforce land use patterns and land cover (LULC) at different levels. However, the statistical information regarding the LULC variations encounters enormous significance for the execution and modelling of appropriate environmental variations and resource management with the available remote sensed data from diverse satellite images and advanced computing technologies; information is generally retrieved from the image classification approaches. However, a broader quantitative analysis of various classification approaches is crucial to choosing an effectual classifier model to acquire appropriate land use regions. We concentrate on the Karavetti region and its related fields in this study. We use a Non-Linear Recurrent Convolutional Neural Network (NLR-CNN) to analyze the data statistically. Well-known techniques such as Support Vector Machine (SVM), Random Forest (RF), and Decision Tree (DT), among others are used to evaluate the model performance. High-resolution images and the data points supplied are also used to assess the accuracy of the categorization and prediction. A confusion matrix is generated where the land cover regions show superior classification accuracy with the fusion model. Also, the NDVI facts and additional metrics like loss, error rate and kappa coefficients are analyzed. Therefore, the outcomes show that the anticipated is considered more robust with better performance to enhance the classification accuracy with the specific land cover regions.

Mitigating network adaptation and QoT prediction challenges in WDM networks

Abstract The capacity and efficiency of optical communication networks have been completely transformed by wavelength division multiplexing (WDM) technology, which allows many wavelengths to be transmitted simultaneously over a single optical fibre. Conventional QoT prediction is based on analytical models that consider physical layer characteristics including dispersion, optical power and signal-to-noise ratio. But these models frequently oversimplify complex real-world situations, which reduces their accuracy for modern high-speed WDM networks. A data-driven solution is provided by machine learning(ML), which may boost the accuracy of QoT predictions by utilising real-time measurements, historical data and a variety of network situations. The creation of a ML-based framework for QoT prediction is investigated in the current research. This research proposes an effective ML-based routing computation model that uses a non-linear autoregressive recurrent neural network (ML-RCNA-RNN) to ensure QoT for every wavelength channel in high-capacity and high-speed WDM networks. Through simulations, more accurate QoT metrics, such as bit error rate (BER) 68.42 %, QoT prediction accuracy (Q-Factor) 5.9 %, network adaption time (ms) 48.3 %, latency (ms) 0.28 % and throughput (Gbps) 14.29 %, have been obtained compared to conventional QoT predictions. These results were obtained using Gaussian noise Python simulation (GNPy). As a result, the proposed framework that makes use of GNPy demonstrates that it substantially enhances optical communication networks’ performance and dependability. This facilitates the development of high-capacity, low-latency and reliable communication infrastructure, and makes it more adaptable and able to manage the complexity of high-speed WDM optical networks while preserving signal quality in the modern digital era.

Nonlinear RNN with noise-immune: A robust and learning-free method for hyperspectral image target detection

While the recurrent neural network (RNN) has achieved remarkable performance on dynamic and control tasks, its applications to image processing, particularly target detection are limited. Challenges arise from differences between the two domains, such as the way for merging time information into static problems and variances of dynamic and static solving methods. To this end, we first extend the existing constrained energy minimization (CEM)-based detection scheme to a dynamic version, e.g. dynamic reinforced CEM (DRCEM), which injects the dynamic information. After that, aided by the rigorous mathematical derivation and optimization theory, the DRCEM is merged into the RNN solution framework. To enhance the robustness and convergence of the existing RNN solutions for improving DRCEM performance, the nonlinear and bounded-constraint RNN (NBCRNN) is designed by developing a novel nonlinear activation function, then applying the proposed model to implement the DRCEM scheme. The corresponding theorem results reveal the proposed model possesses global convergence and enhanced robustness. Compared to state-of-the-art works, the DRCEM solved by the NBCRNN model detection method achieves better detection accuracy, with 1.82% improvement in terms of the Kappa coefficient, and reduces the residual error from 10−4 to 10−7. Furthermore, our detection method is able to preserve the detection accuracy in presence of noise perturbated. To the best of our knowledge, it is the first work to develop the zeroing-type RNN for hyperspectral image target detection. The code and models are publicly available at Github DRCEM_NBCRNN Code Implementation.

A hybrid modelling approach for reverse osmosis processes including fouling

A novel hybrid modelling approach, combining the strengths of a mechanistic reverse osmosis (RO) model and a data-driven fouling model, is developed on a unique long-term dataset from a full-scale RO installation to predict its performance. The mechanistic solution-diffusion model describes well understood phenomena in RO such as concentration polarisation, osmotic pressure and solutes transport throughout the membrane. This solution-diffusion model is combined with a data-driven model to cover the gaps in knowledge related to fouling phenomena. Several fouling models are tested to predict the membrane resistance over time and a thorough analysis of important input features was performed. A non-linear recurrent neural network with long short-term memory (RNN-LSTM) clearly outperformed (RMSE = 1.01e13) a linear autoregressive integrated moving average with exogenous variables (ARIMAX) model (RMSE = 1.97e13) for an 8 month testing period. The best performance was achieved when including membrane cleaning as two separate features representing short and long CIPs. Moreover, recovery setpoint, concentrate flow rate, feed temperature, feed conductivity and feed calcium concentration were shown to be important model input features. Fouling sensitive parameters of the solution-diffusion model (the water permeability, the feed spacer channel height and the solute permeability - determined by a global sensitivity analysis) were made function of the output (membrane resistance) of the RNN-LSTM thus leading to a serial hybrid model. The predictions of the hybrid model showed a clear improvement for all output variables when compared to a solution-diffusion model including temperature correction. The model was developed, calibrated and trained on measurements of standard sensors and can thus be used for real-time applications such as advanced control and predictive scenario analysis in a digital twin context.

A New Prepositioning Technique of a Motion Simulator Platform Using Nonlinear Model Predictive Control and Recurrent Neural Network

The motion cueing algorithm (MCA) is the main algorithm in motion simulators in charge of generating vehicle motions within the platform’s constraints. The classical washout filter is one of the popular types of MCA, which is used in air and land vehicle motion simulators. The fixed home position of the simulator platform is always cogitated in the MCA to washout the motion simulator after generating each motion. Unfortunately, considering the fixed home position reduces the efficient consumption of the workspace in the linear directions. The linear motion of the motion simulator is due to the production of the high-pass frequency part of the motion scenarios. Prepositioning is used to tackle this assumption by varying the home position rather than the fixed position. The linear motion limitations of the motion simulator can virtually be enlarged using the prepositioning method. The efficient regeneration of the high-pass motion cues using a new propositioning technique is the main goal of this study to increase the motion realism of the simulator and remove any false motion cues due to the platform limitations. The proposed model utilised the recurrent neural network (RNN) to estimate the motion scenario along the prediction horizon. The nonlinear model predictive control (MPC) uses the estimated motion signals to extract the best optimal off-centre position of the motion simulator platform. The newly developed prepositioning technique is developed in the simulation environment of MATLAB to validate the proposed technique in terms of efficiency and applicability. The outcomes prove the capability of the proposed technique against the recently developed prepositioning technique using fuzzy logic and RNN.

Synaptic balancing: A biologically plausible local learning rule that provably increases neural network noise robustness without sacrificing task performance.

We introduce a novel, biologically plausible local learning rule that provably increases the robustness of neural dynamics to noise in nonlinear recurrent neural networks with homogeneous nonlinearities. Our learning rule achieves higher noise robustness without sacrificing performance on the task and without requiring any knowledge of the particular task. The plasticity dynamics—an integrable dynamical system operating on the weights of the network—maintains a multiplicity of conserved quantities, most notably the network’s entire temporal map of input to output trajectories. The outcome of our learning rule is a synaptic balancing between the incoming and outgoing synapses of every neuron. This synaptic balancing rule is consistent with many known aspects of experimentally observed heterosynaptic plasticity, and moreover makes new experimentally testable predictions relating plasticity at the incoming and outgoing synapses of individual neurons. Overall, this work provides a novel, practical local learning rule that exactly preserves overall network function and, in doing so, provides new conceptual bridges between the disparate worlds of the neurobiology of heterosynaptic plasticity, the engineering of regularized noise-robust networks, and the mathematics of integrable Lax dynamical systems.

Prognostics for remaining useful life estimation in proton exchange membrane fuel cell by dynamic recurrent neural networks

This paper proposes a nonlinear dynamic recurrent neural network (DRNN) prognostics method for predicting the performance degradation trend and estimating the remaining useful life (RUL) of a proton exchange membrane fuel cell (PEMFC). The conducted DRNN prognostic methods are based on a nonlinear autoregressive neural network (NARNN) model and nonlinear autoregressive with exogenous inputs neural network (NARXNN) model, with the goal of taking actions to extend the life of the PEMFC. To effectively remove data noise or spikes and provide smooth reconstruction, a locally weighted regression (LWR) method is used for pre-processing. The DRNN model parameters for the time lags are processed by an autocorrelation function (ACF). For comparing the predictions within the different DRNN models, the data sets are divided into three different prognostic configurations for predicting and estimating the remaining life of the PEMFC. The results show that the NARXNN model has the best predictions for the degradation trend and accurate remaining useful life, i.e. better than the NARNN model. Finally, the prognostic capability of the proposed DRNN method is compared with those in the literature.

Forecasting the 10.7-cm Solar Radio Flux Using Deep CNN-LSTM Neural Networks

Predicting the time series of 10.7-cm solar radio flux is a challenging task because of its daily variability. This paper proposed a non-linear method, a convolutional and recurrent neural network combined model to achieve end-to-end F10.7 forecasts. The network consists of a one-dimensional convolutional neural network and a long short-term memory network. The CNN network extracted features from F10.7 original data, then trained the feature signals in the long short-term memory network, and outputted the predicted values. The F10.7 daily data during 2003–2014 are used for the testing set. The mean absolute percentage error values of approximately 2.04%, 2.78%, and 4.66% for 1-day, 3-day, and 7-day forecasts, respectively. The statistical results of evaluating the root mean square error, spearman correlation coefficient shows a superior effect as a whole for the 1–27 days forecast, compared with the ordinary single neural network and combination models.

Cluster Forecasting of Corruption Using Nonlinear Autoregressive Models with Exogenous Variables (NARX)—An Artificial Neural Network Analysis

Any effort to combat corruption can benefit from an examination of past and projected worldwide trends. In this paper, we forecast the level of corruption in countries by integrating artificial neural network modeling and time series analysis. The data were obtained from 113 countries from 2007 to 2017. The study is carried out at two levels: (a) the global level, where all countries are considered as a monolithic group; and (b) the cluster level, where countries are placed into groups based on their development-related attributes. For each cluster, we use the findings from our previous study on the cluster analysis of global corruption using machine learning methods that identified the four most influential corruption factors, and we use those as independent variables. Then, using the identified influential factors, we forecast the level of corruption in each cluster using nonlinear autoregressive recurrent neural network models with exogenous inputs (NARX), an artificial neural network technique. The NARX models were developed for each cluster, with an objective function in terms of the Corruption Perceptions Index (CPI). For each model, the optimal neural network is determined by fine-tuning the hyperparameters. The analysis was repeated for all countries as a single group. The accuracy of the models is assessed by comparing the mean square errors (MSEs) of the time series models. The results suggest that the NARX artificial neural network technique yields reliable future values of CPI globally or for each cluster of countries. This can assist policymakers and organizations in assessing the expected efficacies of their current or future corruption control policies from a global perspective as well as for groups of countries.

Soft Computing of a Medically Important Arthropod Vector with Autoregressive Recurrent and Focused Time Delay Artificial Neural Networks.

Simple SummaryArthropod vectors are responsible for transmitting a large number of diseases, and for most, there are still not available effective vaccines. Vector disease control is mostly achieved by a sustained prediction of vector populations to maintain support for surveillance and control activities. Mathematical models may assist in predicting arthropod population dynamics. However, arthropod dynamics, and mosquitoes particularly, due their complex life cycle, often exhibit an abrupt and non-linear occurrence. Therefore, there is a growing interest in describing mosquito population dynamics using new methodologies. In this work, we made an effort to gain insights into the non-linear population dynamics of Culex sp. adults, aiming to introduce straightforward soft-computing techniques based on artificial neural networks (ANNs). We propose two kind of models, one autoregressive, handling temperature as an exogenous driver and population as an endogenous one, and a second based only on the exogenous factor. To the best of our knowledge, this is the first study using recurrent neural networks and the most influential environmental variable for prediction of the WNv vector Culex sp. population dynamics, providing a new framework to be used in arthropod decision-support systems.A central issue of public health strategies is the availability of decision tools to be used in the preventive management of the transmission cycle of vector-borne diseases. In this work, we present, for the first time, a soft system computing modeling approach using two dynamic artificial neural network (ANNs) models to describe and predict the non-linear incidence and time evolution of a medically important mosquito species, Culex sp., in Northern Greece. The first model is an exogenous non-linear autoregressive recurrent neural network (NARX), which is designed to take as inputs the temperature as an exogenous variable and mosquito abundance as endogenous variable. The second model is a focused time-delay neural network (FTD), which takes into account only the temperature variable as input to provide forecasts of the mosquito abundance as the target variable. Both models behaved well considering the non-linear nature of the adult mosquito abundance data. Although, the NARX model predicted slightly better (R = 0.623) compared to the FTD model (R = 0.534), the advantage of the FTD over the NARX neural network model is that it can be applied in the case where past values of the population system, here mosquito abundance, are not available for their forecasting.

Medium-Term Regional Electricity Load Forecasting through Machine Learning and Deep Learning

Due to severe climate change impact on electricity consumption, as well as new trends in smart grids (such as the use of renewable resources and the advent of prosumers and energy commons), medium-term and long-term electricity load forecasting has become a crucial need. Such forecasts are necessary to support the plans and decisions related to the capacity evaluation of centralized and decentralized power generation systems, demand response strategies, and controlling the operation. To address this problem, the main objective of this study is to develop and compare precise district level models for predicting the electrical load demand based on machine learning techniques including support vector machine (SVM) and Random Forest (RF), and deep learning methods such as non-linear auto-regressive exogenous (NARX) neural network and recurrent neural networks (Long Short-Term Memory—LSTM). A dataset including nine years of historical load demand for Bruce County, Ontario, Canada, fused with the climatic information (temperature and wind speed) are used to train the models after completing the preprocessing and cleaning stages. The results show that by employing deep learning, the model could predict the load demand more accurately than SVM and RF, with an R-Squared of about 0.93–0.96 and Mean Absolute Percentage Error (MAPE) of about 4–10%. The model can be used not only by the municipalities as well as utility companies and power distributors in the management and expansion of electricity grids; but also by the households to make decisions on the adoption of home- and district-scale renewable energy technologies.

A novel robust fixed-time convergent zeroing neural network for solving time-varying noise-polluted nonlinear equations

ABSTRACT Solving nonlinear equations is a crucial step in the territories of science and engineering, as many practical problems could be mathematically described by nonlinear equations. In this paper, a novel robust fast convergence zeroing neural network (RFCZNN) by utilizing a reconstructed activation function (AF) is presented and investigated for the dynamic nonlinear equations (DNE) solving problems in predictable period. Comparing with recently reported finite-time nonlinear recurrent neural network, the presented RFCZNN solves the DNE in settled theoretical time and possesses better robustness in noise-polluted environments. Unlike the finite-time convergent neural network models, the time consumption of the presented RFCZNN in convergence process can be calculated directly by mathematics without considering modelling initial states. The comparative experimental results for solving high-order (second and third order) DNE and tracking robotic motional trail are presented separately to further represent that the proposed fixed-time convergent RFCZNN model is more robust and efficient.

Ultra-short-term exogenous forecasting of photovoltaic power production using genetically optimized non-linear auto-regressive recurrent neural networks

Accurate and credible ultra-short-term photovoltaic (PV) power production prediction is very important in short-term resource planning, electric power dispatching, and operational security for the solar power system. This study proposes a novel approach of using genetically optimized non-linear auto-regressive recurrent neural networks (NARX) for ultra-short-term forecasting of PV power output. Hence, the high prediction accuracy of static multi-layered perceptron neural networks can be extended to dynamic (time-series) models with a more stable learning process. Exogenous models with different commonly available meteorological input parameters are developed and tested at five different locations in Algeria and Australia, as case studies of the arid desert climate. The prediction capabilities of the models are quantified as functions of the forecasting horizon (5, 15, 30, and 60 min) and the number of meteorological inputs using various statistical measures. It was found that the proposed models offer very good estimates of output power, with relative root mean square errors ranging between ∼10 and ∼20% and coefficients of determination higher than 91%, while improving the accuracy of corresponding endogenous models by up to 22.3% by only considering the day number and local time as external variables. Unlike the persistent model, the proposed NARX-GA models perform better as the forecasting horizon narrows down, with improvements of up to 58.4%.

A Deep Learning Method for Short-Term Dynamic Positioning Load Forecasting in Maritime Microgrids

The dynamic positioning (DP) system is a progressive technology, which is used in marine vessels and maritime structures. To keep the ship position from displacement in operation mode, its thrusters are used automatically to control and stabilize the position and heading of vessels. Hence, the DP load forecasting is already an essential part of DP vessels, which the DP power demand from the power management system (PMS) for thrusting depends on weather conditions. Furthermore, the PMS is used to control power generation, and prevent power failure, limitation. To perform station keeping of vessels by DPS in environmental changes such as wind, waves, capacity, and reliability of the power generators. Hence, a lack of power may lead to lower DP performance, loss of power, and position, which is called shutdown. Therefore, precise DP power demand prediction for maintaining the vessel position can provide the PMS with sufficient information for better performance in a complex decision-making process for the DP vessel. In this paper, the concept of deep learning techniques is introduced into DPS for DP load forecasting. A Levenberg–Marquardt algorithm based on a nonlinear recurrent neural network is employed in this paper for predicting thrusters’ power consumption in sea state variations due to challenges in power generation with the relative degree of accuracy by combining weather parameter dependencies as environmental disturbances. The proposed method evaluates with three traditional forecasting methods through a set of practical real-time DP load and weather parametric data. Numerical analysis has shown that with the proposed method, the future DP load behavior can be predicted more accurately than that obtained from the traditional methods, which greatly assists in operation and planning of power system to maintain system stability, security, reliability, and economics.

Time-serial analysis of deep neural network models for prediction of climatic conditions inside a greenhouse

Greenhouses provide controlled environmental conditions for crop cultivation but require careful management to ensure ideal growing conditions. In this study, we tested three deep-learning-based neural network models (Artificial neural network, ANN; Nonlinear autoregressive exogenous model, NARX; and Recurrent neural networks – Long short-term memory, RNN-LSTM) to determine the best approach to predicting environmental changes in temperature, humidity, and CO2 within a greenhouse to improve management strategies. This study determined the prediction performance for time steps from 5 to 30 min and showed that the accuracy of the time-based algorithm gradually decreased as prediction time increased. The best model for all datasets was RNN-LSTM, even after 30 min, with an R2 of 0.96 for temperature, 0.80 for humidity, and 0.81 for CO2 concentration. The results of this study show that it is possible to apply deep-learning-based prediction models for more precisely managing greenhouse.

State and fault estimation for nonlinear recurrent neural network systems: Experimental testing on a three‐tank system

AbstractAn observer is presented for the simultaneous estimation of the system state and actuator and sensor faults of a discrete recurrent neural network (RNN) system. The presented approach enables disturbance attenuation and guarantees observer convergence. First, the discrete RNN is converted to a discrete linear parameter varying (LPV) model. Then, the LPV model is further transformed into a descriptor system by extending the system state and sensor fault. Next, an H∞ observer is presented for the simultaneous estimation of the extended state and actuator fault of the descriptor system. Finally, the problem of observer design is translated into solving a linear matrix inequality. Experimental tests on a three‐tank system have validated the effectiveness and correctness of the presented method.

Optimal short-term memory before the edge of chaos in driven random recurrent networks.

The ability of discrete-time nonlinear recurrent neural networks to store time-varying small input signals is investigated with mean-field theory. The combination of a small input strength and mean-field assumptions makes it possible to derive an approximate expression for the conditional probability density of the state of a neuron given a past input signal. From this conditional probability density, we can analytically calculate short-term memory measures, such as memory capacity, mutual information, and Fisher information, and determine the relationships among these measures, which have not been clarified to date to the best of our knowledge. We show that the network contribution of these short-term memory measures peaks before the edge of chaos, where the dynamics of input-driven networks is stable but corresponding systems without input signals are unstable.

Low-Power Sensor Localization in Three-Dimensional Using a Recurrent Neural Network

This letter discusses a novel low-power circuit to self-localize a mobile sensor node in the three-dimensional space using a passive optical receiver. Self-localization of sensors, where a sensor node computes its spatial location by itself, reduces transmission demand, and improves real-time conformity of mobile wireless sensor systems. Our approach forms an over-determined system of angles-of-arrival (AoAs) to mobile sensor received from an optical anchor grid. The AoA system is solved with a linear program (LP) solver, which is implemented using a nonlinear feedback recurrent neural network. To solve the primal and dual LP optimization problems in the AoA system, we show a single multifunctional data path that does not require matrix inversions; thereby, enables within-sensor low-power computations to self-localize. Additionally, unlike other optical indoor positioning architectures, our approach does not require measurements of received signal strength and, thereby, is insensitive to power and alignment imbalances in the anchor grid. We show proof-of-concept field programmable gate array (FPGA) and ASIC simulations of our approach and validate its operation under noisy AoA data and for different numbers of anchors. An FPGA implementation in 180 nm CMOS achieves a peak $\sim$ 0.12 mega localization operations per second per Watt (MOPS/W), while ASIC design in 45 nm CMOS shows a peak $\sim$ 7.7 MOPS/W.

Long-term urban heating load predictions based on optimized retrofit orders: A cross-scenario analysis

District heating technologies are essential key elements of future sustainable energy systems. In order to support the design process, further information concerning long-term developments related to urban heat demand are crucial. Since building refurbishments are indispensable for achieving European CO2 reduction objectives, strategies for district retrofit orders are mandatory, which, in consequence, highly affect future energy demand of urban areas. In this paper, a data-driven approach for predicting long-term urban heating loads with Nonlinear Autoregressive Exogenous Recurrent Neural Networks (NARX RNN) based on an economically optimized retrofit order and two conventional retrofit orders is proposed. For demonstration, measured heat power data of a non-residential district in Germany is used for model training and statistical feature scenario generation enables mapping of future heat demand developments.