Radial Basis Function Nodes Research Articles

Generalized [formula omitted]-sparse algorithms for constructing fault tolerant RBF networks

In the construction process of radial basis function (RBF) networks, two common crucial issues arise: the selection of RBF centers and the effective utilization of the given source without encountering the overfitting problem. Another important issue is the fault tolerant capability. That is, when noise or faults exist in a trained network, it is crucial that the network’s performance does not undergo significant deterioration or decrease. However, without employing a fault tolerant procedure, a trained RBF network may exhibit significantly poor performance. Unfortunately, most existing algorithms are unable to simultaneously address all of the aforementioned issues. This paper proposes fault tolerant training algorithms that can simultaneously select RBF nodes and train RBF output weights. Additionally, our algorithms can directly control the number of RBF nodes in an explicit manner, eliminating the need for a time-consuming procedure to tune the regularization parameter and achieve the target RBF network size. Based on simulation results, our algorithms demonstrate improved test set performance when more RBF nodes are used, effectively utilizing the given source without encountering the overfitting problem. This paper first defines a fault tolerant objective function, which includes a term to suppress the effects of weight faults and weight noise. This term also prevents the issue of overfitting, resulting in better test set performance when more RBF nodes are utilized. With the defined objective function, the training process is designed to solve a generalized M-sparse problem by incorporating an ℓ0-norm constraint. The ℓ0-norm constraint allows us to directly and explicitly control the number of RBF nodes. To address the generalized M-sparse problem, we introduce the noise-resistant iterative hard thresholding (NR-IHT) algorithm. The convergence properties of the NR-IHT algorithm are subsequently discussed theoretically. To further enhance performance, we incorporate the momentum concept into the NR-IHT algorithm, referring to the modified version as “NR-IHT-Mom”. Simulation results show that both the NR-IHT algorithm and the NR-IHT-Mom algorithm outperform several state-of-the-art comparison algorithms.

Using an Opportunity Matrix to Select Centers for RBF Neural Networks

When designed correctly, radial basis function (RBF) neural networks can approximate mathematical functions to any arbitrary degree of precision. Multilayer perceptron (MLP) neural networks are also universal function approximators, but RBF neural networks can often be trained several orders of magnitude more quickly than an MLP network with an equivalent level of function approximation capability. The primary challenge with designing a high-quality RBF neural network is selecting the best values for the network’s “centers”, which can be thought of as geometric locations within the input space. Traditionally, the locations for the RBF nodes’ centers are chosen either through random sampling of the training data or by using k-means clustering. The current paper proposes a new algorithm for selecting the locations of the centers by relying on a structure known as an “opportunity matrix”. The performance of the proposed algorithm is compared against that of the random sampling and k-means clustering methods using a large set of experiments involving both a real-world dataset from the steel industry and a variety of mathematical and statistical functions. The results indicate that the proposed opportunity matrix algorithm is almost always much better at selecting locations for an RBF network’s centers than either of the two traditional techniques, yielding RBF neural networks with superior function approximation capabilities.

Formal convergence analysis on deterministic [formula omitted]-regularization based mini-batch learning for RBF networks

Conventional convergence analysis on mini-batch learning is usually based on the stochastic gradient concept, in which we assume that the training data are presented in a random order. Also, some convergence results require that the learning rate should decrease with the number of training cycles, and that the objective function is a smooth function. Practically speaking, a deterministic presentation scheme with a fixed learning rate is more preferable. Hence, there is a gap between theoretical results and actual implementation. This paper aims at filling the gap. We use the radial basis function (RBF) model for nonlinear regression problems as an example to analyze the convergence properties of mini-batch learning. This paper considers a nonsmooth objective function, which consists of three terms. The coexistence of these three terms is able to handle a number of situations. The first term is a conventional training set error. The second term is a quadratic term which is used to suppress the effect of imperfections in the implementation. The last term is an ℓ1-norm term which is used to select important RBF nodes for the resultant network. Note that the ℓ1-norm term is a nonsmooth function. Although a nonsmooth ℓ1-norm is included and the mini-batch algorithm is deterministic, we are still able to derive the convergence properties, including the sufficient conditions for convergence and range of learning rate. With our results, we have a better theoretical understanding on the behaviour of mini-batch learning and obtain some guidelines on choosing the learning rate. The analysis results can be extended to other flat structural neural network models and other objective functions, which are with quadratic terms and ℓ1-norm.

Evaluation of a radial basis function node refinement algorithm applied to bioheat transfer modeling

Many bioheat transfer problems involve linear/non-linear equations with non-linear or time-dependent boundary conditions. For heat transfer problems, the presence of time and space-dependent functions under Neumann and Mixed type boundary conditions characterize trivial applications in bioengineering, such as thermotherapies, laser surgeries, and burn studies. This greatly increases the complexity of the numerical solution in several problems, requiring fast and accurate numerical solutions. This paper has a main objective evaluate an adaptive mesh refinement radial basis function method strategy for the classical Penne's bioheat transfer modeling. Our numerical results had errors of ~0.1% compared to analytical solutions. Thus, the proposed methodology is accurate and has a low computational cost. For step function heating, two RBF shape parameters were applied, again achieving excellent results. The distributions of the nodes in the solution domain show that the primary source of error in the numerical solutions came from the boundary conditions. This finding should arouse the interest of engineers and scientists in the development of new strategies for problems involving boundary conditions with periodic functions.

Orthogonal least squares based center selection for fault-tolerant RBF networks

In the construction of a radial basis function (RBF) network, one of the most important issues is to determine the number of RBF nodes in the hidden layer and the corresponding centers. However, most existing center selection methods for RBF networks are designed for the faultless implementation situation only. In practice, the implementation of the network may be perturbed by faults and thus fault tolerance should be considered in selecting centers. This paper addresses this problem by focusing on the center selection problem of fault-tolerant RBF networks. In particular, we consider RBF networks under the concurrent fault situation and reformulate the fault-tolerant training objective function as a linear regression problem. A center selection and learning algorithm is then proposed based on the orthogonal least squares (OLS) algorithm. The proposed algorithm takes fault-tolerance into consideration both in center selection and network training, and it is able to determine the number of RBF nodes, the RBF centers and the trained connecting weights simultaneously. Simulation results show that the proposed algorithm is superior than state-of-the-art center selection algorithms.

An $\ell_0$ -Norm-Based Centers Selection for Failure Tolerant RBF Networks

There are two important issues in the construction of a radial basis function (RBF) neural network. The first one is to select suitable RBF centers. The second one is that the resultant RBF network should be with good fault tolerance. This paper proposes an algorithm that is able to select RBF centers and to train fault tolerant RBF networks simultaneously. The proposed algorithm borrows the concept from sparse approximation. In our formulation, we first define a fault tolerant objective function based on all input vectors from the training samples. We then introduce the minimax concave penalty (MCP) function, which is an approximation of l 0 -norm, into the objective function. The MCP term is able to force some unimportant RBF weights to zero. Hence the RBF node selection process can be achieved during training. As the MCP function is nondifferentiable and nonconvex, traditional gradient descent based algorithms are still unable to minimize the modified objective function. Based on the alternating direction method of multipliers (ADMM) framework, we develop an algorithm, called ADMM-MCP, to minimize the modified objective function. The convergent proof of the proposed ADMM-MCP algorithm is also presented. Simulation results show that the proposed ADMM-MCP algorithm is superior to many existing center selection algorithms under the concurrent fault situation.

Ensemble decision forest of RBF networks via hybrid feature clustering approach for high-dimensional data classification

Classification of the high-dimensional data is challenging due to the curse of dimensionality, heavy computational burden and decreasing precision of algorithms. In order to mitigate these effects, feature selection approaches that can determine an efficient subset of features are utilized in the processing. However, most of these techniques attain just one subset of non-redundant features including the best ones. Alternatively, clustering approaches can be used to find the most informative clusters of features instead of generating just a single subset. So called, Hybrid Feature Clustering (HFC) method is capable of maximizing the classification accuracy while keeping the amount of redundant features in each cluster low. The patterns of each cluster are classified by a neural tree that employs Radial Basis Function (RBF) for the nodes. Within each neural tree, a hierarchical approach is proposed to transfer the knowledge of synaptic weights from a parent RBF node to each child. A gating network is applied on the forest of these neural trees in order to aggregate the results. By assessing the classification accuracy and the computational complexity on high-dimensional datasets it can be shown that the proposed solution has outperformed the state of the art classifiers. Furthermore, the computational complexity and the convergence of this method are theoretically proven and the robustness analysis under noisy conditions is conducted.

LCA based RBF training algorithm for the concurrent fault situation

In the construction of a radial basis function (RBF) network, one of the most important issues is the selection of RBF centers. However, many selection methods are designed for the fault free situation only. This paper first assumes that all the training samples are used for constructing a fault tolerant RBF network. We then add an l1 norm regularizer into the fault tolerant objective function. According to the nature of the l1 norm regularizer, some unnecessary RBF nodes are removed automatically during training. Based on the local competition algorithm (LCA) concept, we propose an analog method, namely fault tolerant LCA (FTLCA), to minimize the fault tolerant objective function. We prove that the proposed fault tolerant objective function has a unique optimal solution, and that the FTLCA converges to the global optimal solution. Simulation results show that the FTLCA is better than the orthogonal least square approach and the support vector regression approach.

Online training and its convergence for faulty networks with multiplicative weight noise

A recent article showed that the objective function of the online weight noise injection algorithm is not equal to the training set error of faulty radial basis function (RBF) networks under the weight noise situation (Ho et al., 2010). Hence the online weight noise injection algorithm is not able to optimize the training set error of faulty networks with multiplicative weight noise. This paper proposes an online learning algorithm to tolerate multiplicative weight noise. Two learning rate cases, namely fixed learning rate and adaptive learning rate, are investigated. For the fixed learning rate case, we show that if the learning rate μ is less than 2/(σb2+maxi‖ϕ(xi)|2), then the online algorithm converges, where xi׳s are the training input vectors, σ2b is the variance of the multiplicative weight noise, ϕ(xi)=[ϕ1(xi),…,ϕM(xi)]T, and ϕj(·) is the output of the j-th RBF node. In addition, as μ→0, the trained weight vector tends to the optimal solution. For the adaptive learning rate case, let the learning rates {μk} be a decreasing sequence and limk→∞μk=0, where k is the index of learning cycles. We prove that if ∑k=1∞μk=∞ and ∑k=1∞μk2<∞, then the weight vector converges to the optimal solution. Our simulation results show that the performance of the proposed algorithm is better than that of the conventional online approaches, such as the online weight decay and weight noise injection.

Domain Decomposition Combined Radial Basis Function Collocation Method to Solve Transient Eddy Current Magnetic Problems With Moving Conductors

Radial basis function (RBF) collocation method is a kind of pure meshless numerical method and has been applied to solve static and transient electromagnetic problems. Especially, it shows a great advantage in the computation of moving conductor eddy current magnetic problems. To simulate the conductor movement, the field equations are decoupled with superposition principle and solved by time-domain iteration under moving coordinate systems. One problem is that the coefficient matrix of RBF governing equations, which needs to be computed in each iteration step, is full. As the number of RBF nodes increases, the computational capacity will grow rapidly. The domain decomposition method (DDM), which divides the solving domain into several subdomains, could be conveniently combined with RBF collocation method. This paper first applies DDM combined RBF collocation method to compute transient eddy current magnetic field problems with moving conductors. With this novel method, the iteration only proceeds in the subdomains containing conductors. And the magnetic field in the subdomains without conductors needs to be computed just once before the iteration. The dimension of the coefficient matrix computed in the iteration is only determined by the number of nodes in the corresponding subdomains and on the interfaces. An engineering problem is computed to show that the DDM combined RBF collocation method is much more efficient than the normal one.

Particle Swarm Optimization Aided Orthogonal Forward Regression for Unified Data Modeling

We propose a unified data modeling approach that is equally applicable to supervised regression and classification applications, as well as to unsupervised probability density function estimation. A particle swarm optimization (PSO) aided orthogonal forward regression (OFR) algorithm based on leave-one-out (LOO) criteria is developed to construct parsimonious radial basis function (RBF) networks with tunable nodes. Each stage of the construction process determines the center vector and diagonal covariance matrix of one RBF node by minimizing the LOO statistics. For regression applications, the LOO criterion is chosen to be the LOO mean square error, while the LOO misclassification rate is adopted in two-class classification applications. By adopting the Parzen window estimate as the desired response, the unsupervised density estimation problem is transformed into a constrained regression problem. This PSO aided OFR algorithm for tunable-node RBF networks is capable of constructing very parsimonious RBF models that generalize well, and our analysis and experimental results demonstrate that the algorithm is computationally even simpler than the efficient regularization assisted orthogonal least square algorithm based on LOO criteria for selecting fixed-node RBF models. Another significant advantage of the proposed learning procedure is that it does not have learning hyperparameters that have to be tuned using costly cross validation. The effectiveness of the proposed PSO aided OFR construction procedure is illustrated using several examples taken from regression and classification, as well as density estimation applications.

Research on an online self-organizing radial basis function neural network

A new growing and pruning algorithm is proposed for radial basis function (RBF) neural network structure design in this paper, which is named as self-organizing RBF (SORBF). The structure of the RBF neural network is introduced in this paper first, and then the growing and pruning algorithm is used to design the structure of the RBF neural network automatically. The growing and pruning approach is based on the radius of the receptive field of the RBF nodes. Meanwhile, the parameters adjusting algorithms are proposed for the whole RBF neural network. The performance of the proposed method is evaluated through functions approximation and dynamic system identification. Then, the method is used to capture the biochemical oxygen demand (BOD) concentration in a wastewater treatment system. Experimental results show that the proposed method is efficient for network structure optimization, and it achieves better performance than some of the existing algorithms.

Non-linear system identification using particle swarm optimisation tuned radial basis function models

A novel particle swarm optimisation (PSO) tuned radial basis function (RBF) network model is proposed for identification of non-linear systems. At each stage of orthogonal forward regression (OFR) model construction process, PSO is adopted to tune one RBF unit's centre vector and diagonal covariance matrix by minimising the leave-one-out (LOO) mean square error (MSE). This PSO aided OFR automatically determines how many tunable RBF nodes are sufficient for modelling. Compared with the-state-of-the-art local regularisation assisted orthogonal least squares algorithm based on the LOO MSE criterion for constructing fixed-node RBF network models, the PSO tuned RBF model construction produces more parsimonious RBF models with better generalisation performance and is often more efficient in model construction. The effectiveness of the proposed PSO aided OFR algorithm for constructing tunable node RBF models is demonstrated using three real data sets.

Construction of Tunable Radial Basis Function Networks Using Orthogonal Forward Selection

An orthogonal forward selection (OFS) algorithm based on leave-one-out (LOO) criteria is proposed for the construction of radial basis function (RBF) networks with tunable nodes. Each stage of the construction process determines an RBF node, namely, its center vector and diagonal covariance matrix, by minimizing the LOO statistics. For regression application, the LOO criterion is chosen to be the LOO mean-square error, while the LOO misclassification rate is adopted in two-class classification application. This OFS-LOO algorithm is computationally efficient, and it is capable of constructing parsimonious RBF networks that generalize well. Moreover, the proposed algorithm is fully automatic, and the user does not need to specify a termination criterion for the construction process. The effectiveness of the proposed RBF network construction procedure is demonstrated using examples taken from both regression and classification applications.

A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks

In this paper, we develop an online sequential learning algorithm for single hidden layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes in a unified framework. The algorithm is referred to as online sequential extreme learning machine (OS-ELM) and can learn data one-by-one or chunk-by-chunk (a block of data) with fixed or varying chunk size. The activation functions for additive nodes in OS-ELM can be any bounded nonconstant piecewise continuous functions and the activation functions for RBF nodes can be any integrable piecewise continuous functions. In OS-ELM, the parameters of hidden nodes (the input weights and biases of additive nodes or the centers and impact factors of RBF nodes) are randomly selected and the output weights are analytically determined based on the sequentially arriving data. The algorithm uses the ideas of ELM of Huang et al. developed for batch learning which has been shown to be extremely fast with generalization performance better than other batch training methods. Apart from selecting the number of hidden nodes, no other control parameters have to be manually chosen. Detailed performance comparison of OS-ELM is done with other popular sequential learning algorithms on benchmark problems drawn from the regression, classification and time series prediction areas. The results show that the OS-ELM is faster than the other sequential algorithms and produces better generalization performance.

Using the correlation criterion to position and shape RBF units for incremental modelling

A novel technique is proposed for the incremental construction of sparse radial basis function (RBF) networks. The correlation between a RBF regressor and the training data is used as the criterion to position and shape the RBF node, and it is shown that this is equivalent to incrementally minimise the modelling mean square error. A guided random search optimisation method, called the repeated weighted boosting search, is adopted to append RBF nodes one by one in an incremental regression modelling procedure. The experimental results obtained using the proposed method demonstrate that it provides a viable alternative to the existing state-of-the-art modelling techniques for constructing parsimonious RBF models that generalise well.

Universal approximation using incremental constructive feedforward networks with random hidden nodes.

According to conventional neural network theories, single-hidden-layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes are universal approximators when all the parameters of the networks are allowed adjustable. However, as observed in most neural network implementations, tuning all the parameters of the networks may cause learning complicated and inefficient, and it may be difficult to train networks with nondifferential activation functions such as threshold networks. Unlike conventional neural network theories, this paper proves in an incremental constructive method that in order to let SLFNs work as universal approximators, one may simply randomly choose hidden nodes and then only need to adjust the output weights linking the hidden layer and the output layer. In such SLFNs implementations, the activation functions for additive nodes can be any bounded nonconstant piecewise continuous functions g : R --> R and the activation functions for RBF nodes can be any integrable piecewise continuous functions g : R --> R and integral of R g(x)dx not equal to 0. The proposed incremental method is efficient not only for SFLNs with continuous (including nondifferentiable) activation functions but also for SLFNs with piecewise continuous (such as threshold) activation functions. Compared to other popular methods such a new network is fully automatic and users need not intervene the learning process by manually tuning control parameters.

On the training of radial basis function classifiers

An approach for the implementation of the Radial Basis Function (RBF) technique is presented and applied to a network of the appropriate architecture. The paper explores a methodology for selecting kernel function parameters and the extent to which the number of RBF nodes can be reduced without significantly affecting the overall training error. These objectives are accomplished through algorithms that shall be described in detail. Emphasis is also placed on the problems faced by a technique that has been proved superior to the more traditional training algorithms, particularly in terms of processing speed and solvability of nonlinear patterns. Solutions are consequently proposed in view of making RBF a more efficient method for interpolation and classification purposes.