Nonlinear Recurrent Neural Network Research Articles

Cloud based multicasting using fat tree data confidential recurrent neural network

ABSTRACT With the progress of cloud computing, more users are attracted by its strong and cost-effective computation potentiality. Nevertheless, whether Cloud Service Providers can efficiently protect Cloud Users data confidentiality (DC) remains a demanding issue. The CU may execute several applications with multicast needs. In Cloud different techniques were used to provide DC with multicast necessities. In this work, we aim at ensuring DC in the cloud. This is achieved using a two-step technique, called Fat Tree Data Confidential Recurrent Neural Network (FT-DCRNN) in a cloud environment. The first step performs the construction of Fat Tree based on Multicast model. The aim to use Fat Tree with Multicast model is that the multicast model propagates traffic on multiple links. With the Degree Restrict Multicast Fat Tree construction algorithm using a reference function, the minimum average between two links is measured. With these measured links, multicast is said to be performed that in turn improves the throughput and efficiency of cloud service. Then, with the objective of providing DC for the multi-casted data or messages, DCRNN model is applied. With the Non-linear Recurrent Neural Network using Logistic Activation Function, by handling complex non-linear relationships, average response time is said to be reduced.

Convolutional Nonlinear Differential Recurrent Neural Networks for Crowd Scene Understanding

With the growth of crowd phenomena in the real world, crowd scene understanding is becoming an important task in anomaly detection and public security. Visual ambiguities and occlusions, high density, low mobility, and scene semantics, however, make this problem a great challenge. In this paper, we propose an end-to-end deep architecture, convolutional nonlinear differential recurrent neural networks (CNDRNNs), for crowd scene understanding. CNDRNNs consist of GoogleNet Inception V3 convolutional neural networks (CNNs) and nonlinear differential recurrent neural networks (RNNs). Different from traditional non-end-to-end solutions which separate the steps of feature extraction and parameter learning, CNDRNN utilizes a unified deep model to optimize the parameters of CNN and RNN hand in hand. It thus has the potential of generating a more harmonious model. The proposed architecture takes sequential raw image data as input, and does not rely on tracklet or trajectory detection. It thus has clear advantages over the traditional flow-based and trajectory-based methods, especially in challenging crowd scenarios of high density and low mobility. Taking advantage of CNN and RNN, CNDRNN can effectively analyze the crowd semantics. Specifically, CNN is good at modeling the semantic crowd scene information. On the other hand, nonlinear differential RNN models the motion information. The individual and increasing orders of derivative of states (DoS) in differential RNN can progressively build up the ability of the long short-term memory (LSTM) gates to detect different levels of salient dynamical patterns in deeper stacked layers modeling higher orders of DoS. Lastly, existing LSTM-based crowd scene solutions explore deep temporal information and are claimed to be “deep in time.” Our proposed method CNDRNN, however, models the spatial and temporal information in a unified architecture and achieves “deep in space and time.” Extensive performance studies on the Violent-Flows, CUHK Crowd, and NUS-HGA datasets show that the proposed technique significantly outperforms state-of-the-art methods.

A novel recurrent neural network and its finite-time solution to time-varying complex matrix inversion

A complex-valued nonlinear recurrent neural network is designed and researched for time-varying matrix inversion solving in complex field. Unlike the design methods of the conventional gradient neural network (CGNN) and the previous Zhang neural network (ZNN), the proposed complex-valued nonlinear recurrent neural network (CVNRNN) model is established on basis of a nonlinear evolution formula and possesses a better finite-time convergence Besides, the detailed theoretical analysis provides a guarantee for the finite-time convergence achievement of the CVNRNN model. In addition, the theoretical analysis is also verified by numerical simulations, which comparatively show that the proposed CVNRNN model is faster and more accurate than the ZNN model and the CGNN model in solving time-varying complex matrix inversion.

Data-driven heating and cooling load predictions for non-residential buildings based on support vector machine regression and NARX Recurrent Neural Network: A comparative study on district scale

Predicting building energy consumption is essential for planning and managing energy systems. In recent times, numerous studies focus on load forecasting models dealing with a wide range of different methods. In addition to Artificial Neural Networks (ANN), especially Support Vector Machines (SVM) have been studied. Various research work showed the success and superiority of ANN and SVM for load predictions, where frequently, SVM outperformed ANN models. In this study, data-driven thermal load forecasting performance of ε-SVM Regression (ε-SVM-R) based on a Radial Basis Function (RBF) and a polynomial kernel is compared to the outcome of two Nonlinear Autoregressive Exogenous Recurrent Neural Networks (NARX RNN) of different depths. For demonstration, historical data from a non-residential district in Germany is used for training and testing to predict monthly loads. The evaluation of the resulting predictions show that NARX RNNs yields higher accuracy than (ε-SVM-R) models, in combination with comparable computational effort.

Local stability analysis and H∞ performance for Lipschitz digital filters with saturation nonlinearity and external interferences

Abstract This paper proposes a novel method to analyze the local stability of Lipschitz nonlinear digital filtering schemes under saturation overflow nonlinearity. Conditions for the stability analysis and robust performance estimation are provided in the form of matrix inequalities by utilizing Lyapunov theory, local saturation overflow arithmetic, and Lipschitz condition. The proposed criterion ascertains (local) asymptotic stability in the absence of perturbations. Under the effects of external interferences, a condition for the local stability, ensuring the H∞ performance objective, is developed. The proposed approach offers a less conservative and more accurate estimate of H∞ performance index than the global method by utilizing a bound on the interferences energy. Moreover, the proposed criterion, in contrast to the existing global methods, can be employed to choose an adequate word length of a digital hardware for the specified values of tolerable perturbations energy, H∞ performance index, and fixed-point resolution. It is worth mentioning that analysis approaches have not been completely reported in the literature, in which local stability criteria for nonlinear discrete-time filtering prototypes under both overflow and disturbances have been developed. A detailed stability analysis for a nonlinear recurrent neural network is performed for demonstrating the effectiveness of the proposed scheme.

Stabilized supralinear network can give rise to bistable, oscillatory, and persistent activity

A hallmark of cortical circuits is their versatility. They can perform multiple fundamental computations such as normalization, memory storage, and rhythm generation. Yet it is far from clear how such versatility can be achieved in a single circuit, given that specialized models are often needed to replicate each computation. Here, we show that the stabilized supralinear network (SSN) model, which was originally proposed for sensory integration phenomena such as contrast invariance, normalization, and surround suppression, can give rise to dynamic cortical features of working memory, persistent activity, and rhythm generation. We study the SSN model analytically and uncover regimes where it can provide a substrate for working memory by supporting two stable steady states. Furthermore, we prove that the SSN model can sustain finite firing rates following input withdrawal and present an exact connectivity condition for such persistent activity. In addition, we show that the SSN model can undergo a supercritical Hopf bifurcation and generate global oscillations. Based on the SSN model, we outline the synaptic and neuronal mechanisms underlying computational versatility of cortical circuits. Our work shows that the SSN is an exactly solvable nonlinear recurrent neural network model that could pave the way for a unified theory of cortical function.

Recurrent Neuro-Fuzzy Control of Grid-Interfaced Solid Oxide Fuel Cell (SOFC) System

The paper presents controller design, modelling, and simulations of the solid oxide fuel cell (SOFC) system. The SOFC model is used for the development of the fuzzy control scheme to improve the system performance. The SOFC is widely acknowledged for the clean distributed power generation. However, dire process problems occur frequently when the stand-alone fuel cell is directly interfaced with the electricity grid. Moreover, sustaining the optimal power quality and load following is the huge challenge, during the peak power demand schedule from the utility grid and large load perturbations. Consequently, a suitable and highly efficient control system is required for controlling and following the power load demands for the complex grid interfaced SOFC power systems. Therefore, a novel nonlinear hybrid adaptive recurrent fuzzy neural network (ARFNN) is developed for the control of the grid interfaced SOFC. The rapid power load following and safe SOFC operation requirement is also considered in the design of the closed loop control system. Simulation results proved that the proposed hybrid ARFNN enhances the optimal power quality and load-following than the conventional PI control scheme.

Nonlinear recurrent neural networks for finite-time solution of general time-varying linear matrix equations

In order to solve general time-varying linear matrix equations (LMEs) more efficiently, this paper proposes two nonlinear recurrent neural networks based on two nonlinear activation functions. According to Lyapunov theory, such two nonlinear recurrent neural networks are proved to be convergent within finite-time. Besides, by solving differential equation, the upper bounds of the finite convergence time are determined analytically. Compared with existing recurrent neural networks, the proposed two nonlinear recurrent neural networks have a better convergence property (i.e., the upper bound is lower), and thus the accurate solutions of general time-varying LMEs can be obtained with less time. At last, various different situations have been considered by setting different coefficient matrices of general time-varying LMEs and a great variety of computer simulations (including the application to robot manipulators) have been conducted to validate the better finite-time convergence of the proposed two nonlinear recurrent neural networks.

A hybrid solar radiation modeling approach using wavelet multiresolution analysis and artificial neural networks

Assessment of solar potential over a location of interest is an important step towards the successful planning of renewable energy projects. However, solar data are not available for every point of interest due to the absence of meteorological stations and sophisticated solar sensors, so solar radiation has to be estimated using models. This paper presents a hybrid technique to improve the performance of a widely used modeling technique i.e. artificial neural network (ANN). Four different architectures of ANN, namely: multilayer perceptron (MLP), Adaptive neuro-fuzzy inference system (ANFIS), Nonlinear autoregressive recurrent exogenous neural network (NARX), and generalized regression neural networks (GRNN), are used in this study. A wavelet multiresolution analysis is applied to decompose the complex meteorological signals into relatively simple parts, wavelet sub-series, using discrete wavelet transformation (DWT) algorithm. The wavelet sub-series are modeled by the ANN models and reconstructed to estimate the original signal. Hence, enhancing the learning process of these models. Four meteorological parameters, namely: temperature (T), relative humidity (RH), wind speed (WS), and sunshine duration (SSD), are used to mode the global horizontal irradiation (GHI) over Abu Dhabi, the United Arab Emirates. The proposed approach is compared to standalone ANN models and validated using well-known statistical validation metrics including coefficient of determination (R2), root mean square error (RMSE), mean bias error (MBE), mean absolute percentage error (MAPE), and t-statistics. In addition, wavelet cross spectrum (WCS) is used as a visual indicator of the model performance in time, frequency, and phase domains. The results show that using the proposed strategy considerably improves the modeling performance of the ANN with a maximum improvement of 6.84% in R2 for MLP. In addition, minimum RMSE of 2.78% is observed for GRNN.

Non‐linear recurrent ANN‐based LFC design considering the new structures of Q matrix

This paper presents the design of a non-linear recurrent artificial neural network (ANN) based load frequency control (LFC) of a two-area power system interconnected via HVDC tie-line in parallel with EHVAC line. The control design is based on the combination of robust LFC and optimisation of recurrent ANN so that designed LFC scheme is suitable to handle the diverse operating conditions of power system. The various structures of the state cost weighting matrix ( Q ), i.e. controllability and observability aspects which affect the dynamics of the power system are used for the LFC design. The feedback gains achieved by the implementation of robust LFC using the different structures of Q are used to effectively train the recurrent ANN-based LFC design and its performance is analysed with and without system non-linearity's for 1% step load disturbance in one of the control areas to show the superiority of one design over the others. The closed-loop system eigenvalues obtained for the system ensure the system stability. Furthermore, the recurrent ANN-based LFC performance is evaluated for the diverse system operating conditions and compared with multi-layer perceptron (MLP) ANN-based and conventional PI control schemes to demonstrate the effectiveness of proposed LFC design scheme.

Motor-Skill Learning in an Insect Inspired Neuro-Computational Control System.

In nature, insects show impressive adaptation and learning capabilities. The proposed computational model takes inspiration from specific structures of the insect brain: after proposing key hypotheses on the direct involvement of the mushroom bodies (MBs) and on their neural organization, we developed a new architecture for motor learning to be applied in insect-like walking robots. The proposed model is a nonlinear control system based on spiking neurons. MBs are modeled as a nonlinear recurrent spiking neural network (SNN) with novel characteristics, able to memorize time evolutions of key parameters of the neural motor controller, so that existing motor primitives can be improved. The adopted control scheme enables the structure to efficiently cope with goal-oriented behavioral motor tasks. Here, a six-legged structure, showing a steady-state exponentially stable locomotion pattern, is exposed to the need of learning new motor skills: moving through the environment, the structure is able to modulate motor commands and implements an obstacle climbing procedure. Experimental results on a simulated hexapod robot are reported; they are obtained in a dynamic simulation environment and the robot mimicks the structures of Drosophila melanogaster.

General Recurrent Neural Network for Solving Generalized Linear Matrix Equation

This brief proposes a general framework of the nonlinear recurrent neural network for solving online the generalized linear matrix equation (GLME) with global convergence property. If the linear activation function is utilized, the neural state matrix of the nonlinear recurrent neural network can globally and exponentially converge to the unique theoretical solution of GLME. Additionally, as compared with the case of using the linear activation function, two specific types of nonlinear activation functions are proposed for the general nonlinear recurrent neural network model to achieve superior convergence. Illustrative examples are shown to demonstrate the efficacy of the general nonlinear recurrent neural network model and its superior convergence when activated by the aforementioned nonlinear activation functions.

A modular architecture for transparent computation in recurrent neural networks

Computation is classically studied in terms of automata, formal languages and algorithms; yet, the relation between neural dynamics and symbolic representations and operations is still unclear in traditional eliminative connectionism. Therefore, we suggest a unique perspective on this central issue, to which we would like to refer as transparent connectionism, by proposing accounts of how symbolic computation can be implemented in neural substrates. In this study we first introduce a new model of dynamics on a symbolic space, the versatile shift, showing that it supports the real-time simulation of a range of automata. We then show that the Gödelization of versatile shifts defines nonlinear dynamical automata, dynamical systems evolving on a vectorial space. Finally, we present a mapping between nonlinear dynamical automata and recurrent artificial neural networks. The mapping defines an architecture characterized by its granular modularity, where data, symbolic operations and their control are not only distinguishable in activation space, but also spatially localizable in the network itself, while maintaining a distributed encoding of symbolic representations. The resulting networks simulate automata in real-time and are programmed directly, in the absence of network training. To discuss the unique characteristics of the architecture and their consequences, we present two examples: (i) the design of a Central Pattern Generator from a finite-state locomotive controller, and (ii) the creation of a network simulating a system of interactive automata that supports the parsing of garden-path sentences as investigated in psycholinguistics experiments.

Hourly global solar irradiation forecasting for New Zealand

With the growing use of solar thermal energy systems and small scale photovoltaic power generation by domestic users, there is increasing need to develop intelligent controllers that allow these users to efficiently manage the energy generated by these systems. Ideally these intelligent controllers will be able to forecast the availability and magnitude of the solar resource to plan in advance for periods when the solar irradiance magnitude is small or unavailable. In addition, the method used to provide this forecast needs to be adaptable to a range of timescales and locations. With this in mind, this study examined the possibility of providing a 24-h ahead forecast of hourly global solar irradiation in New Zealand using several approaches but with particular reference to nonlinear autoregressive recurrent neural networks with exogenous inputs (NARX).Hourly time series data for nine historic weather variables recorded over a three year period was used to train and test the forecasting methods for New Zealand’s largest city, Auckland. Results from forecasts based on the NARX were compared with an artificial neural network (ANN) based Multilayer Perceptron (MLP) method, a statistical approach using auto regressive moving average (ARMA) and a reference persistence approach. Predicted values of hourly global solar irradiation were compared with the measured values, and it was found that the root mean squared error (RMSE) was 0.243MJ/m2 for the NARX method as compared to 0.484MJ/m2, 0.315MJ/m2 and 0.514MJ/m2 for the MLP, ARMA and persistence approaches respectively. Subsequently the NARX approach was used to forecast global solar irradiation for other major cities across New Zealand. The results demonstrate the ability of the NARX approach to forecast irradiation values at a later time and across a number of different locations. As such it is foreseeable that such an approach could serve as the basis of a forecasting system in future intelligent controllers.

Stimulus statistics shape oscillations in nonlinear recurrent neural networks.

Rhythmic activity plays a central role in neural computations and brain functions ranging from homeostasis to attention, as well as in neurological and neuropsychiatric disorders. Despite this pervasiveness, little is known about the mechanisms whereby the frequency and power of oscillatory activity are modulated, and how they reflect the inputs received by neurons. Numerous studies have reported input-dependent fluctuations in peak frequency and power (as well as couplings across these features). However, it remains unresolved what mediates these spectral shifts among neural populations. Extending previous findings regarding stochastic nonlinear systems and experimental observations, we provide analytical insights regarding oscillatory responses of neural populations to stimulation from either endogenous or exogenous origins. Using a deceptively simple yet sparse and randomly connected network of neurons, we show how spiking inputs can reliably modulate the peak frequency and power expressed by synchronous neural populations without any changes in circuitry. Our results reveal that a generic, non-nonlinear and input-induced mechanism can robustly mediate these spectral fluctuations, and thus provide a framework in which inputs to the neurons bidirectionally regulate both the frequency and power expressed by synchronous populations. Theoretical and computational analysis of the ensuing spectral fluctuations was found to reflect the underlying dynamics of the input stimuli driving the neurons. Our results provide insights regarding a generic mechanism supporting spectral transitions observed across cortical networks and spanning multiple frequency bands.

Nonlinear Recurrent Neural Network Predictive Control for Energy Distribution of a Fuel Cell Powered Robot

This paper presents a neural network predictive control strategy to optimize power distribution for a fuel cell/ultracapacitor hybrid power system of a robot. We model the nonlinear power system by employing time variant auto-regressive moving average with exogenous (ARMAX), and using recurrent neural network to represent the complicated coefficients of the ARMAX model. Because the dynamic of the system is viewed as operating- state- dependent time varying local linear behavior in this frame, a linear constrained model predictive control algorithm is developed to optimize the power splitting between the fuel cell and ultracapacitor. The proposed algorithm significantly simplifies implementation of the controller and can handle multiple constraints, such as limiting substantial fluctuation of fuel cell current. Experiment and simulation results demonstrate that the control strategy can optimally split power between the fuel cell and ultracapacitor, limit the change rate of the fuel cell current, and so as to extend the lifetime of the fuel cell.

Complex-valued pipelined decision feedback recurrent neural network for non-linear channel equalisation

A novel complex-valued non-linear equaliser-based pipelined decision feedback recurrent neural network (CPDFRNN) is proposed in this study for non-linear channel equalisation in wireless communication systems. The CPDFRNN with low computational complexity, a modular structure comprising a number of modules that are interconnected in a chained form, is an extension of the recently proposed real-valued pipelined decision feedback recurrent neural equalisers. Each module is implemented by a small-scale complex-valued decision feedback recurrent neural network (CDFRNN). Moreover, a decision feedback part in each module can overcome the unstable characteristic of the complex-valued recurrent neural network (CRNN). To suit the modularity of the CPDFRNN, an adaptive amplitude complex-valued real-time recurrent learning (CRTRL) algorithm is presented. Simulations demonstrate that the CPDFRNN equaliser using the amplitude CRTRL algorithm with less computational complexity not only eliminates the adverse effects of the nesting architecture, but also provides a superior performance over the CRNN and CDFRNN equalisers for non-linear channels in wireless communication systems.

Nonlinear enhancement of noisy speech, using continuous attractor dynamics formed in recurrent neural networks

Here, formation of continuous attractor dynamics in a nonlinear recurrent neural network is used to achieve a nonlinear speech denoising method, in order to implement robust phoneme recognition and information retrieval. Formation of attractor dynamics in recurrent neural network is first carried out by training the clean speech subspace as the continuous attractor. Then, it is used to recognize noisy speech with both stationary and nonstationary noise. In this work, the efficiency of a nonlinear feedforward network is compared to the same one with a recurrent connection in its hidden layer. The structure and training of this recurrent connection, is designed in such a way that the network learns to denoise the signal step by step, using properties of attractors it has formed, along with phone recognition. Using these connections, the recognition accuracy is improved 21% for the stationary signal and 14% for the nonstationary one with 0db SNR, in respect to a reference model which is a feedforward neural network.

An Assessment of Qualitative Performance of Machine Learning Architectures: Modular Feedback Networks

A framework for the assessment of qualitative performance of machine learning architectures is proposed. For generality, the analysis is provided for the modular nonlinear pipelined recurrent neural network (PRNN) architecture. This is supported by a sensitivity analysis, which is achieved based upon the prediction performance with respect to changes in the nature of the processed signal and by utilizing the recently introduced delay vector variance (DVV) method for phase space signal characterization. Comprehensive simulations combining the quantitative and qualitative analysis on both linear and nonlinear signals suggest that better quantitative prediction performance may need to be traded in order to preserve the nature of the processed signal, especially where the signal nature is of primary importance (biomedical applications).

Application of a Recurrent Neural Network to Space Deversity in SDMA and CDMA Mobile Communication Systems

Linear and non-linear adaptive algorithms are investigated for Space Division Multiple Access (SDMA). SDMA is one of the emerging techniques for multiple access of users in mobile radio, which uses spatial distribution of users for their differentiation. The performance of the linear Square Root Kalman (SRK) algorithm for SDMA is compared to that of the non-linear Recurrent Neural Network (RNN) technique. The proposed SDMA-RNN technique is evaluated over Rician fading channels, and it shows improved Bit Error Rate (BER) performance in comparison with the linear SRK-based technique. The performance of SDMA-RNN is also compared with that of Code Division Multiple Access (CDMA) systems, showing that it could be used as a viable alternative scheme for multiple access of users. Finally, a Hybrid CDMA-SDMA system is proposed combining CDMA and SDMA-RNN systems. Hybrid CDMA-SDMA exhibits a very good potential for increase in the capacity and the performance of mobile communications systems.