Pipelined Recurrent Neural Network Research Articles

Online-Growing Neural Network Control for Dissolved Oxygen Concentration

The activated sludge method is one of the most commonly used methods for wastewater treatment process (WWTP). Among them, the dissolved oxygen (DO) concentration is a key factor affecting microbial metabolism and wastewater treatment effectiveness. However, due to the nonlinearity and dynamicity of WWTP, it is difficult to precisely control the DO concentration by traditional control methods. To solve this problem, the online-growing pipelined recurrent wavelet neural network (OG-PRWNN) control is proposed to improve the DO control accuracy. First, the online growing mechanism is designed to adjust the number of modules of the controller by measuring the control performance. Then, the structure of the controller is automatically determined to meet the different operating conditions of the WWTP. Second, the online algorithm of parameters incorporating adaptive learning rates is designed to train the OG-PRWNN to meet the control requirements. In addition, the stability of the OG-PRWNN controller is analyzed by the Lyapunov stability theorem. Finally, the performance of the controller is verified by the benchmark simulation model of WWTP. Simulation results show that the OG-PRWNN controller can obtain better control accuracy.

Self-organizing pipelined recurrent wavelet neural network for time series prediction

The pipelined recurrent wavelet neural network (PRWNN) with nested structure has been widely used for time series prediction. However, selecting the appropriate network structure for different problems is still an unsolved problem. To solve this issue, the self-organizing PRWNN (SPRWNN) is proposed. Different from the PRWNN with fixed structure, the number of modules and hidden neurons of SPRWNN can be automatically determined during the training process. Firstly, the number of hidden neurons in each module of the SPRWNN is adjusted by the spiking strength (SS) of nodes. Then, the module growth mechanism is designed by measuring the performance of the SPRWNN. Furthermore, the convergence of SPRWNN is proved theoretically. The experimental results show that the SPRWNN can not only automatically adjust the network structure, but also improves the prediction accuracy by about 40% as compared with the PRWNN with fixed structure.

A novel pipelined neural FIR architecture for nonlinear adaptive filter

This paper presents a novel adaptive pipelined neural finite impulse response (PNFIR) filter for nonlinear signal processing. Unlike traditional pipelined recurrent neural network (PRNN), each module of the PNFIR filter is a simple architecture that includes a standard FIR filter followed by a nonlinear activation function. The complete design of proposed filter includes two subsections: The nonlinear part consists of neural FIR (NFIR) modules which is interconnected in a chained form and simultaneously executed in a parallel fashion; the linear subsection is a tapped-delay-line (TDL) linear combiner. Based on convex combination architecture, the adaptive algorithm derived from the gradient descent approach is utilized to update weights of the nonlinear and linear parts. Moreover, the analysis of stability conditions and computational complexity is also presented. Numerous simulation experimental results on nonlinear dynamic systems identification, speech signal and chaotic time series prediction show that the proposed PNFIR filter has simpler architecture, faster convergence rate, and lower computation complexity than the PRNN and joint process filter using pipelined feedforward second-order Volterra architecture (JPPSOV).

Complex-Valued Pipelined Chebyshev Functional Link Recurrent Neural Network for Joint Compensation of Wideband Transmitter Distortions and Impairments

In this article, a novel digital predistortion (DPD) model based on complex-valued pipelined Chebyshev functional link recurrent neural network (CPCFLRNN) for joint compensation of wideband transmitter distortions and impairments is proposed. The functional link neural network (FLNN) model has attracted much attention from scholars, and many improved models using this structure, such as Chebyshev FLNN, have been applied in the DPD of power amplifiers (PAs). However, these existing neural network models cannot deal with complex-valued input signals simultaneously, and the real-valued model structure will introduce cumbersome training algorithm and result in a long training time. The pipelined recurrent neural network (PRNN) has been successfully applied to nonlinear signal prediction because of its excellent ability for dealing with nonlinear nonstationary signals. Therefore, the PRNN model containing Chebyshev structure is extended to complex domain for the first time to obtain the CPCFLRNN model for DPD application. Considering the strong correlation of in-phase and quadrature phase (I/Q) components of the transmitter signal, the real time recurrent learning (RTRL) algorithm based on fully complex activation function is selected and extended to complex domain to obtain the complex-valued RTRL (CRTRL) algorithm for CPCFLRNN model training. A GaN PA was employed to verify the effectiveness of the proposed models. And the input signal is a 30MHz LTE signals which consists of I/Q imbalance and dc offsets. The experimental results show that the proposed CPCFLRNN model have more accurate modeling effect and better linearization performance compared with the conventional DPD models.

Using adaptive neural networks to provide self-healing autonomic software

Self-managing systems are a significant feature in autonomic computing; required for a system's reliability and performance in a changing environment. The work described in this paper is concerned with self-healing systems; systems that can detect and analyse issues with their behaviour and performance, and enact fixes or reconfigurations as appropriate. These processes should occur in a real-time manner, restoring desired functionality as soon as possible. The system should ideally maintain functionality during the healing process, occurring at runtime. Neural networks, specifically recursive networks, are proposed as a solution to some of these challenges; monitoring the system and environment, mapping a suitable resolution and adapting the system accordingly. A novel application of a modified pipelined recurrent neural network is proposed in this paper with experiments aimed to assess the applicability to online, at runtime, adaptation with multiple input variables. The first set of experiments applies a pipelined recurrent neural network to control server requests across a bank of servers, routing and adding/removing capacity to maximise efficiency and ensure service quality. The second experiments supplement an overcurrent relay with a pipelined recurrent neural network, monitoring, and if necessary, replicating its circuit breaking functionality as required based on line signal measurements.

Identification of Nonlinear Dynamic System Using a Novel Recurrent Wavelet Neural Network Based on the Pipelined Architecture

This paper presents a novel modular recurrent neural network based on the pipelined architecture (PRWNN) to reduce the computational complexity and improve the performance of the recurrent wavelet neural network (RWNN). The PRWNN inherits the modular architectures of the pipelined recurrent neural network proposed by Haykin and Li and is made up of a number of RWNN modules that are interconnected in a chained form. Since those modules of the PRWNN can be simultaneously performed in a pipelined parallelism fashion, this would lead to a crucial improvement of computational efficiency. Furthermore, owing to the cascade interconnection of dynamic modules, the performance of the PRWNN can be further enhanced. An adaptive gradient algorithm based on the real-time recurrent learning is derived to suit for the modular PRWNN. Simulation examples are given to evaluate the effectiveness of the PRWNN model on the identification of nonlinear dynamic systems and analysis of sunspot number time series. According to simulation results, it is clearly shown that the PRWNN provides impressive better performance in comparison with the single RWNN model.

A novel joint-processing adaptive nonlinear equalizer using a modular recurrent neural network for chaotic communication systems

To eliminate nonlinear channel distortion in chaotic communication systems, a novel joint-processing adaptive nonlinear equalizer based on a pipelined recurrent neural network (JPRNN) is proposed, using a modified real-time recurrent learning (RTRL) algorithm. Furthermore, an adaptive amplitude RTRL algorithm is adopted to overcome the deteriorating effect introduced by the nesting process. Computer simulations illustrate that the proposed equalizer outperforms the pipelined recurrent neural network (PRNN) and recurrent neural network (RNN) equalizers.

Pipelined Chebyshev Functional Link Artificial Recurrent Neural Network for Nonlinear Adaptive Filter

A novel nonlinear adaptive filter with pipelined Chebyshev functional link artificial recurrent neural network (PCFLARNN) is presented in this paper, which uses a modification real-time recurrent learning algorithm. The PCFLARNN consists of a number of simple small-scale Chebyshev functional link artificial recurrent neural network (CFLARNN) modules. Compared to the standard recurrent neural network (RNN), those modules of PCFLARNN can simultaneously be performed in a pipelined parallelism fashion, and this would lead to a significant improvement in its total computational efficiency. Furthermore, contrasted with the architecture of a pipelined RNN (PRNN), each module of PCFLARNN is a CFLARNN whose nonlinearity is introduced by enhancing the input pattern with Chebyshev functional expansion, whereas the RNN of each module in PRNN utilizing linear input and first-order recurrent term only fails to utilize the high-order terms of inputs. Therefore, the performance of PCFLARNN can further be improved at the cost of a slightly increased computational complexity. In addition, due to the introduced nonlinear functional expansion of each module in PRNN, the number of input signals can be reduced. Computer simulations have demonstrated that the proposed filter performs better than PRNN and RNN for nonlinear colored signal prediction, nonstationary speech signal prediction, and chaotic time series prediction.

A novel nonlinear adaptive filter using a pipelined second-order Volterra recurrent neural network

To enhance the performance and overcome the heavy computational complexity of recurrent neural networks (RNN), a novel nonlinear adaptive filter based on a pipelined second-order Volterra recurrent neural network (PSOVRNN) is proposed in this paper. A modified real-time recurrent learning (RTRL) algorithm of the proposed filter is derived in much more detail. The PSOVRNN comprises of a number of simple small-scale second-order Volterra recurrent neural network (SOVRNN) modules. In contrast to the standard RNN, these modules of a PSOVRNN can be performed simultaneously in a pipelined parallelism fashion, which can lead to a significant improvement in its total computational efficiency. Moreover, since each module of the PSOVRNN is a SOVRNN in which nonlinearity is introduced by the recursive second-order Volterra (RSOV) expansion, its performance can be further improved. Computer simulations have demonstrated that the PSOVRNN performs better than the pipelined recurrent neural network (PRNN) and RNN for nonlinear colored signals prediction and nonlinear channel equalization. However, the superiority of the PSOVRNN over the PRNN is at the cost of increasing computational complexity due to the introduced nonlinear expansion of each module.

An Assessment of Qualitative Performance of Machine Learning Architectures: Modular Feedback Networks

A framework for the assessment of qualitative performance of machine learning architectures is proposed. For generality, the analysis is provided for the modular nonlinear pipelined recurrent neural network (PRNN) architecture. This is supported by a sensitivity analysis, which is achieved based upon the prediction performance with respect to changes in the nature of the processed signal and by utilizing the recently introduced delay vector variance (DVV) method for phase space signal characterization. Comprehensive simulations combining the quantitative and qualitative analysis on both linear and nonlinear signals suggest that better quantitative prediction performance may need to be traded in order to preserve the nature of the processed signal, especially where the signal nature is of primary importance (biomedical applications).

Nonlinear adaptive prediction of complex-valued signals by complex-valued PRNN

A complex-valued pipelined recurrent neural network (CPRNN) for nonlinear adaptive prediction of complex nonlinear and nonstationary signals is introduced. This architecture represents an extension of the recently proposed real-valued PRNN of Haykin and Li in 1995. To train the CPRNN, a complex-valued real time recurrent learning (CRTRL) algorithm is first derived for a single recurrent neural network (RNN). This algorithm is shown to be generic and applicable to general signals that have complex domain representations. The CRTRL is then extended to suit the modularity of the CPRNN architecture. Further, to cater to the possibly large dynamics of the input signals, a gradient adaptive amplitude of the nonlinearity within the neurons is introduced to give the adaptive amplitude CRTRL (AACRTRL). A comprehensive analysis of the architecture and associated learning algorithms is undertaken, including the role of the number of nested modules, number of neurons within the modules, and input memory of the CPRNN. Simulations on real-world and synthetic complex data support the proposed architecture and algorithms.

On the choice of parameters of the cost function in nested modular RNN's

We address the choice of the coefficients in the cost function of a modular nested recurrent neural-network (RNN) architecture, known as the pipelined recurrent neural network (PRNN). Such a network can cope with the problem of vanishing gradient, experienced in prediction with RNN's. Constraints on the coefficients of the cost function, in the form of a vector norm, are considered. Unlike the previous cost function for the PRNN, which included a forgetting factor motivated by the recursive least squares (RLS) strategy, the proposed forms of cost function provide "forgetting" of the outputs of adjacent modules based upon the network architecture. Such an approach takes into account the number of modules in the PRNN, through the unit norm constraint on the coefficients of the cost function of the PRNN. This is shown to be particularly suitable, since due to inherent nesting in the PRNN, every module gives its full contribution to the learning process, whereas the unit norm constrained cost function introduces a sense of forgetting in the memory management of the PRNN. The PRNN based upon a modified cost function outperforms existing PRNN schemes in the time series prediction simulations presented.

Advanced RNN Based NARMA Predictors

An analysis of nonlinear time series prediction schemes, realised though advanced Recurrent Neural Network (RNN) techniques is provided. Due to practical constraints in using common RNNs, such as the problem of vanishing gradient, some other ways to improve RNN based prediction are analysed. This is undertaken for a simple RNN through to the Pipelined Recurrent Neural Network (PRNN), which consists of a number of nested small-scale RNNs. A Nonlinear AutoRegressive Moving Average (NARMA) nonlinear model is introduced in the context of RNN architectures, and an posteriori mode of operation within that framework. Moreover, it is shown that the basic a priori PRNN structure exhibits certain a posteriori features. The PRNN based predictor, is shown to exhibit nesting, and to be able to represent block cascaded stochastic models, such as the Wiener–Hammerstein model. Simulations undertaken on a speech signal support the analysis.

Toward an optimal PRNN-based nonlinear predictor

We present an approach for selecting optimal parameters for the pipelined recurrent neural network (PRNN) in the paradigm of nonlinear and nonstationary signal prediction. Although there has recently been progress in terms of algorithms for training the PRNN, no account has been made of some inherent features of the PRNN. We therefore provide a study of the role of nesting, which is inherent to the PRNN architecture. The corresponding number of nested modules needed for a certain prediction task, and their contribution toward the final prediction gain (PG) give a thorough insight into the way the PRNN performs, and offers solutions for optimization of its parameters. In particular, nesting, which is a contractive function by its nature, allows the forgetting factor in the cost function of the PRNN to exceed unity, hence it becomes an emphasis factor. This compensates for the small contribution of the distant modules to the prediction process, due to nesting, and helps to circumvent the problem of vanishing gradient, experienced in RNN's for prediction. The PRNN, with its parameters chosen based upon the established criteria, is shown to outperform the linear least mean square (LMS) and recursive least squares (RLS) predictors, as well as previously proposed PRNN schemes, at no expense of additional computational complexity.

Nonlinear adaptive prediction of speech with a pipelined recurrent neural network

New learning algorithms for an adaptive nonlinear forward predictor that is based on a pipelined recurrent neural network (PRNN) are presented. A computationally efficient gradient descent (GD) learning algorithm, together with a novel extended recursive least squares (ERLS) learning algorithm, are proposed. Simulation studies based on three speech signals that have been made public and are available on the World Wide Web (WWW) are used to test the nonlinear predictor. The gradient descent algorithm is shown to yield poor performance in terms of prediction error gain, whereas consistently improved results are achieved with the ERLS algorithm. The merit of the nonlinear predictor structure is confirmed by yielding approximately 2 dB higher prediction gain than a linear structure predictor that employs the conventional recursive least squares (RLS) algorithm.

Optimal nonlinear adaptive prediction and modeling of MPEG video in ATM networks using pipelined recurrent neural networks

This paper investigates the application of a pipelined recurrent neural network (PRNN) to the adaptive traffic prediction of MPEG video signal via dynamic ATM networks. The traffic signal of each picture type (I, P, and B) of MPEG video is characterized by a general nonlinear autoregressive moving average (NARMA) process. Moreover, a minimum mean-squared error predictor based on the NARMA model is developed to provide the best prediction for the video traffic signal. However, the explicit functional expression of the best mean-squared error predictor is actually unknown. To tackle this difficulty, a PRNN that consists of a number of simpler small-scale recurrent neural network (RNN) modules with less computational complexity is conducted to introduce the best nonlinear approximation capability into the minimum mean-squared error predictor model in order to accurately predict the future behavior of MPEG video traffic in a relatively short time period based on adaptive learning for each module from previous measurement data, in order to provide faster and more accurate control action to avoid the effects of excessive load situation. Since those modules of PRNN can be performed simultaneously in a pipelined parallelism fashion, this would lead to a significant improvement in the total computational efficiency of PRNN. In order to further improve the convergence performance of the adaptive algorithm for PRNN, a learning-rate annealing schedule is proposed to accelerate the adaptive learning process. Another advantage of the PRNN-based predictor is its generalization from learning that is useful for learning a dynamic environment for MPEG video traffic prediction in ATM networks where observations may be incomplete, delayed, or partially available. The PRNN-based predictor presented in this paper is shown to be promising and practically feasible in obtaining the best adaptive prediction of real-time MPEG video traffic.

Nonlinear adaptive prediction of nonstationary signals

We describe a computationally efficient scheme for the nonlinear adaptive prediction of nonstationary signals whose generation is governed by a nonlinear dynamical mechanism. The complete predictor consists of two subsections. One performs a nonlinear mapping from the input space to an intermediate space with the aim of linearizing the input signal, and the other performs a linear mapping from the new space to the output space. The nonlinear subsection consists of a pipelined recurrent neural network (PRNN), and the linear section consists of a conventional tapped-delay-line (TDL) filter. The nonlinear adaptive predictor described is of general application. The dynamic behavior of the predictor is demonstrated for the case of a speech signal; for this application, it is shown that the nonlinear adaptive predictor outperforms the traditional linear adaptive scheme in a significant way.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>