Networks For Function Approximation Research Articles

A growing and pruning sequential learning algorithm of hyper basis function neural network for function approximation

Radial basis function (RBF) neural network is constructed of certain number of RBF neurons, and these networks are among the most used neural networks for modeling of various nonlinear problems in engineering. Conventional RBF neuron is usually based on Gaussian type of activation function with single width for each activation function. This feature restricts neuron performance for modeling the complex nonlinear problems. To accommodate limitation of a single scale, this paper presents neural network with similar but yet different activation function—hyper basis function (HBF). The HBF allows different scaling of input dimensions to provide better generalization property when dealing with complex nonlinear problems in engineering practice. The HBF is based on generalization of Gaussian type of neuron that applies Mahalanobis-like distance as a distance metrics between input training sample and prototype vector. Compared to the RBF, the HBF neuron has more parameters to optimize, but HBF neural network needs less number of HBF neurons to memorize relationship between input and output sets in order to achieve good generalization property. However, recent research results of HBF neural network performance have shown that optimal way of constructing this type of neural network is needed; this paper addresses this issue and modifies sequential learning algorithm for HBF neural network that exploits the concept of neuron’s significance and allows growing and pruning of HBF neuron during learning process. Extensive experimental study shows that HBF neural network, trained with developed learning algorithm, achieves lower prediction error and more compact neural network.

Using Function Approximation to Determine Neural Network Accuracy

Many, if not most, control processes demonstrate non-linear behavior in some portion of their operating range and the ability of neural networks to model non-linear dynamics makes them very appealing for control. Control of high reliability safety systems, and autonomous control in process or robotic applications, however, require accurate and consistent control and neural networks are only approximators of various functions so their degree of approximation becomes important. In this paper, the factors affecting the ability of a feed-forward back-propagation neural network to accurately approximate a non-linear function are explored. Compared to pattern recognition using a neural network for function approximation provides an easy and accurate method for determining the network's accuracy. In contrast to other techniques, we show that errors arising in function approximation or curve fitting are caused by the neural network itself rather than scatter in the data. A method is proposed that provides improvements in the accuracy achieved during training and resulting ability of the network to generalize after training. Binary input vectors provided a more accurate model than with scalar inputs and retraining using a small number of the outlier x,y pairs improved generalization.

Digital Implementation of Artificial Neural Network for Function Approximation and Pressure Control Applications

The soft computing algorithms are being nowadays used for various multi input multi output complicated non linear control applications. This paper presented the development and implementation of back propagation of multilayer perceptron architecture developed in FPGA using VHDL. The usage of the FPGA (Field Programmable Gate Array) for neural network implementation provides flexibility in programmable systems. For the neural network based instrument prototype in real time application. The conventional specific VLSI neural chip design suffers the limitation in time and cost. With low precision artificial neural network design, FPGA have higher speed and smaller size for real time application than the VLSI design. The challenges are finding an architecture that minimizes the hardware cost, maximizing the performance, accuracy. The goal of this work is to realize the hardware implementation of neural network using FPGA. Digital system architecture is presented using Very High Speed Integrated Circuits Hardware Description Language (VHDL)and is implemented in FPGA chip. MATLAB ANN programming and tools are used for training the ANN. The trained weights are stored in different RAM, and is implemented in FPGA. The design was tested on a FPGA demo board. Keywords- Backpropagation, field programmable gate array (FPGA) hardware implementation, multilayer perceptron, pressure sensor, Xilinx FPGA.

Neural Networks for Computational Chemistry: Pitfalls and Recommendations

ABSTRACTThere is a long history of using neural networks for function approximation in computational physics and chemistry. Despite their conceptual simplicity, the practitioner may face difficulties when it comes to putting them to work. This small guide intends to pinpoint some neural networks pitfalls, along with corresponding solutions to successfully realize function approximation tasks in physics, chemistry or other fields.

Investigation of Neural Networks for Function Approximation

In this work, some ubiquitous neural networks are applied to model the landscape of a known problem function approximation. The performance of the various neural networks is analyzed and validated via some well-known benchmark problems as target functions, such as Sphere, Rastrigin, and Griewank functions. The experimental results show that among the three neural networks tested, Radial Basis Function (RBF) neural network is superior in terms of speed and accuracy for function approximation in comparison with Back Propagation (BP) and Generalized Regression Neural Network (GRNN).

Using Radial Basis Function Networks for Function Approximation and Classification

The radial basis function (RBF) network has its foundation in the conventional approximation theory. It has the capability of universal approximation. The RBF network is a popular alternative to the well-known multilayer perceptron (MLP), since it has a simpler structure and a much faster training process. In this paper, we give a comprehensive survey on the RBF network and its learning. Many aspects associated with the RBF network, such as network structure, universal approimation capability, radial basis functions, RBF network learning, structure optimization, normalized RBF networks, application to dynamic system modeling, and nonlinear complex-valued signal processing, are described. We also compare the features and capability of the two models.

Modified wavelet neural network in function approximation and its application in prediction of time-series pollution data

Properly designing a wavelet neural network (WNN) is crucial for achieving the optimal generalization performance. In this paper, in order to improve the predictive capability of WNNs, the types of activation functions used in the hidden layer of the WNN were varied. The modified WNNs were then applied in approximating a benchmark piecewise function. Subsequently, performance comparisons with other developed methods in studying the same benchmark function were made. An assessment analysis showed that this proposed approach outperformed the rest. The efficiency of the modified WNNs was explored through a real-world application problem-specifically, the prediction of time-series pollution data at Texas of United States. The comparative experimental results showed that integrating different wavelet families into the hidden layer of WNNs leads to superior performance.

Adaptive Learning Control for Finite Interval Tracking Based on Constructive Function Approximation and Wavelet

Using a constructive function approximation network, an adaptive learning control (ALC) approach is proposed for finite interval tracking problems. The constructive function approximation network consists of a set of bases, and the number of bases can evolve when learning repeats. The nature of the basis allows the continuous adaptive learning of parameters when the network undergoes any structural changes, and consequently offers the flexibility in tuning the network structure. The expandability of the bases guarantees precision of the function approximation and avoids the trial-and-error procedure in structure selection for any fixed structure network. Two classes of unknown nonlinear functions, namely, either global L(2) or local L(2) with a known bounding function, are taken into consideration. Using the Lyapunov method, the existence of solution and the convergence property of the proposed ALC system are discussed in a rigorous manner. By virtue of the celebrated orthonormal and multiresolution properties, wavelet network is used as the universal function approximator, with the weights tuned by the proposed adaptive learning mechanism.

Hardware implementation of pulse mode RBF neural network-based image denoising

In this paper, we propose a very compact implementation of a pulse mode radial basis activation function (RBF). The main idea is to make use of the powerful means of RBF neural networks in function approximation and implement a reconfigurable hardware, achieving different image processing tasks with on-chip learning. The proposed network is applied here as illustration in image denoising, which is a very important step in image processing. The efficiency of the proposed pulse mode RBF neural network in image denoising versus other conventional filtering techniques is demonstrated. Moreover, it has been shown that the best strategy, leading to better learning generalisation performances, is to apply in the learning steps all kinds of noise in a random way. In such a way, generalisation is better with respect to each kind of noise. The corresponding design was implemented on a Virtex II PRO FPGA platform and synthesis results are presented.

Design of Hybrid Fuzzy Neural Network for Function Approximation

In this paper, a hybrid Fuzzy Neural Network (FNN) system for function approximation is presented. The proposed FNN can handle numeric and fuzzy inputs simultaneously. The numeric inputs are fuzzified by input nodes upon presentation to the network while the Fuzzy rule based knowledge is translated directly into network architecture. The connections between input to hidden nodes represent rule antecedents and hidden to output nodes represent rule consequents. All the connections are represented by Gaussian fuzzy sets. The method of activation spread in the network is based on a fuzzy mutual subsethood measure. Rule (hidden) node activations are computed as a fuzzy inner product. For a given numeric o fuzzy input, numeric outputs are computed using volume based defuzzification. A supervised learning procedure based on gradient descent is employed to train the network. The model has been tested on two different approximation problems: sine-cosine function approximation and Narazaki-Ralescu function and shows its natural capability of inference, function approximation, and classification.

A systematic investigation of a neural network for function approximation

A model which takes advantage of wavelet-like functions in the functional form of a neural network is used for function approximation. The scale parameters are mainly used, neglecting the usual translation parameters in the function expansion. Two training operations are then investigated. The first one consists of optimizing the output synaptic weights and the second one on optimizing the scale parameters hidden inside the elementary tasks. Building upon previously published results, it is found that if (p+1) scale parameters merge during the learning process, derivatives of order p will emerge spontaneously in the functional basis. It is also found that for those tasks which induce such mergings, the function approximation can be improved and the training time reduced by directly implementing the elementary tasks and their derivatives in the functional basis. Attention has been also devoted to the role transfer functions, number of iterations, and formal neurons number may play during and after the learning process. The results complement previously published results on this problem.

Dynamic Modeling of Nonlinear Systems Using Wavelet Networks

The purpose of this paper is to present wavelet networks as a new scheme to model nonlinear systems. The capabilities of wavelet networks in function approximation make them appealing for system modeling. The wavelet networks presented are utilized in the dynamic modeling of a nonlinear servomechanism. A new wavelet network scheme is proposed for the identification of the nonlinearity in the servomechanism. Simulation results show the modeling performance of both schemes.

Rbf Neural Networks for Function Approximation in Dynamic Modelling

Rbf Neural Networks for Function Approximation in Dynamic ModellingThe paper demonstrates the comparison of Monte Carlo simulation algorithm with neural network enhancement in the reliability case study. With regard to process dynamics, we attempt to evaluate the tank system unreliability related to the initiative input parameters setting. The neural network is used in equation coefficients calculation, which is executed in each transient state. Due to the neural networks, for some of the initial component settings we can achieve the results of computation faster than in classical way of coefficients calculating and substituting into the equation.

Combining neural networks for function approximation under conditions of sparse data: the biased regression approach

In most real world applications when large amounts of data are sparse, constructing a neural network or a committee network to achieve good performance is very difficult. This paper reports on research that attempts to improve generalization capabilities of committee networks by investigating new committee fusers under sparse data conditions. A bias/variance decomposition is performed to gain insights into the outcome. The committees are formed based on a linear combination of neural networks using the concepts of ridge regression (RR), principal component regression (PCR), and the r–k class estimator. Here, a set of trained bootstrap networks serve as an input variable and the target response is the dependent variable in the regression models. In this paper, a new algorithm—the minimum error estimate (MEE) is developed to tune the parameters of the regression models so that the committees will be combined to achieve better generalization. The experimental results demonstrate that the best of the proposed methods perform at least as well as the baseline methods under all sparse data conditions. By automatically selecting the tuning parameters, the proposed algorithms can integrate unique information from the committee members as well as effectively reduce multicollinearity effects, and thus provide reasonable performances.

A max-piecewise-linear neural network for function approximation

This paper proposes a Max-Piecewise-Linear (MPWL) Neural Network for function approximation. The MPWL network consists of a single hidden layer and employs the Piecewise-Linear (PWL) Basis Functions as the activation functions of hidden neurons. Since a PWL Basis Function possesses a simple functional form and universal representation capability, the MPWL network achieves a good balance between the computational simplicity and approximation accuracy. In addition, a PWL version of Back-Propagation (PBP) algorithm is developed, whose computational complexity is lower than the training algorithms for the Canonical PWL network, and the Back-Propagation algorithm for the sigmoid network with same number of training cycles. Another advantage of the MPWL network is its amenability to hardware implementation. This facilitates many applications such as nonlinear circuit synthesis, dynamic identification and control.

On global-local artificial neural networks for function approximation.

We present a hybrid radial basis function (RBF) sigmoid neural network with a three-step training algorithm that utilizes both global search and gradient descent training. The algorithm used is intended to identify global features of an input-output relationship before adding local detail to the approximating function. It aims to achieve efficient function approximation through the separate identification of aspects of a relationship that are expressed universally from those that vary only within particular regions of the input space. We test the effectiveness of our method using five regression tasks; four use synthetic datasets while the last problem uses real-world data on the wave overtopping of seawalls. It is shown that the hybrid architecture is often superior to architectures containing neurons of a single type in several ways: lower mean square errors are often achievable using fewer hidden neurons and with less need for regularization. Our global-local artificial neural network (GL-ANN) is also seen to compare favorably with both perceptron radial basis net and regression tree derived RBFs. A number of issues concerning the training of GL-ANNs are discussed: the use of regularization, the inclusion of a gradient descent optimization step, the choice of RBF spreads, model selection, and the development of appropriate stopping criteria.

Function approximation using generalized adalines

This paper proposes neural organization of generalized adalines (gadalines) for data driven function approximation. By generalizing the threshold function of adalines, we achieve the K-state transfer function of gadalines which responds a unitary vector of K binary values to the projection of a predictor on a receptive field. A generative component that uses the K-state activation of a gadaline to trigger K posterior independent normal variables is employed to emulate stochastic predictor-oriented target generation. The fitness of a generative component to a set of paired data mathematically translates to a mixed integer and linear programming. Since consisting of continuous and discrete variables, the mathematical framework is resolved by a hybrid of the mean field annealing and gradient descent methods. Following the leave-one-out learning strategy, the obtained learning method is extended for optimizing multiple generative components. The learning result leads to parameters of a deterministic gadaline network for function approximation. Numerical simulations further test the proposed learning method with paired data oriented from a variety of target functions. The result shows that the proposed learning method outperforms the MLP and RBF learning methods for data driven function approximation.

A Generalized Growing and Pruning RBF (GGAP-RBF) Neural Network for Function Approximation

This paper presents a new sequential learning algorithm for radial basis function (RBF) networks referred to as generalized growing and pruning algorithm for RBF (GGAP-RBF). The paper first introduces the concept of significance for the hidden neurons and then uses it in the learning algorithm to realize parsimonious networks. The growing and pruning strategy of GGAP-RBF is based on linking the required learning accuracy with the significance of the nearest or intentionally added new neuron. Significance of a neuron is a measure of the average information content of that neuron. The GGAP-RBF algorithm can be used for any arbitrary sampling density for training samples and is derived from a rigorous statistical point of view. Simulation results for bench mark problems in the function approximation area show that the GGAP-RBF outperforms several other sequential learning algorithms in terms of learning speed, network size and generalization performance regardless of the sampling density function of the training data.

Performance comparison between the training method and the numerical method of the orthogonal neural network in function approximation

The orthogonal neural network is a recently developed neural network based on the properties of orthogonal functions. It can avoid the drawbacks of traditional feedforward neural networks such as initial values of weights, number of processing elements, and slow convergence speed. Nevertheless, it needs many processing elements if a small training error is desired. Therefore, numerous data sets are required to train the orthogonal neural network. In the article, a least-squares method is proposed to determine the exact weights by applying limited data sets. By using the Lagrange interpolation method, the desired data sets required to solve for the exact weights can be calculated. An experiment in approximating typical continuous and discrete functions is given. The Chebyshev polynomial is chosen to generate the processing elements of the orthogonal neural network. The experimental results show that the numerical method in determining the weights gives as good performance in approximation error as the known training method and the former has less convergence time. © 2004 Wiley Periodicals, Inc. Int J Int Syst 19: 1257–1275, 2004.

Automatic training of a min-max neural network for function approximation by using a second feed forward network

This research is concerned with a gradient descent training algorithm of a min-max network which we will refer to as the target network. Training makes use of a helper feed-forward network (FFN) to represent the performance function used in training the target network. A helper FFN is trained because the mathematical form of the performance function for the target network in terms of its trainable parameters, p, is not differentiable. Values for the parameter vector, p, of the target network are generated randomly and performance values are determined to produce the data for training the helper FFN with its own connection matrices. Thus we find an approximation to the mathematical relationship between performance values and p by training an FFN. The input to this FFN is a value for p and the output is a performance measure. The transfer function of the helper FFN provides a differentiable function for the performance function of the parameter vector, p, for the target network allowing gradient search methods for finding the optimum p for the target network. The method is successfully tried in approximating a given function and also on training data produced by a randomly selected min-max network.