Generalization Ability Of Neural Networks Research Articles

Enhancing the generalization ability of neural networks through controlling the hidden layers

In this paper we proposed two new variants of backpropagation algorithm. The common point of these two new algorithms is that the outputs of nodes in the hidden layers are controlled with the aim to solve the moving target problem and the distributed weights problem. One algorithm ( AlgoRobust) is not so insensitive to the noises in the data, the second one ( AlgoGS) is through using Gauss–Schmidt algorithm to determine in each epoch which weight should be updated, while the other weights are kept unchanged in this epoch. In this way a better generalization can be obtained. Some theoretical explanations are also provided. In addition, simulation comparisons are made between Gaussian regularizer, optimal brain damage (OBD) and the proposed algorithms. Simulation results confirm that the new proposed algorithms perform better than that of Gaussian regularizer, and the first algorithm AlgoRobust performs better than the second algorithm AlgoGS in the noisy data. On the other hand AlgoGS performs better than the AlgoRobust on the data without noise and the final structure obtained by two new algorithms is comparable to that obtained by using OBD.

Dynamic pruning algorithm for multilayer perceptron based neural control systems

Generalization ability of neural networks is very important and a rule of thumb for good generalization in neural systems is that the smallest system should be used to fit the training data. Unfortunately, it is normally difficult to determine the optimal size of networks, particularly, in the sequential training applications such as online control. In this paper, an online training algorithm with a dynamic pruning procedure is proposed for the online tuning and pruning the neural tracking control system. The conic sector theory is introduced in the design of this robust neural control system, which aims at providing guaranteed boundedness for both the input–output signals and the weights of the neural network. The proposed algorithm is applied to a multilayer perceptron with adjustable weights and a complete convergence proof is provided. The neural control system guarantees the closed-loop stability of the estimation, and in turn, a good tracking performance. The performance improvement of the proposed system over existing systems can be qualified in terms of better generalization ability, preventing weight shifts, fast convergence and robustness against system disturbance.

Bayesian Neural Network Analysis of Undrained Side Resistance of Drilled Shafts

A Bayesian neural network algorithm was used to model the relationship between the soil undrained shear strength, the effective overburden stress, and the undrained side resistance alpha factor for drilled shafts. The database comprised 127 field load tests. The Bayesian approach to learning in neural networks is a recent enhancement to improve the generalization ability of neural networks. Instead of just giving a single “optimum” prediction, the Bayesian approach provides information on the characteristic error of the prediction that arises from the uncertainty associated with interpolating noisy data. The developed neural network model provides good estimates of the undrained side resistance adhesion factor. Parametric studies using the trained neural network model suggest that the effective overburden stress directly or indirectly has an influence on the adhesion factor for drilled shafts. One distinct benefit of this neural network model is the computation of the error bars on the predictions of the ...

Testing the generalization of artificial neural networks with cross-validation and independent-validation in modelling rice tillering dynamics

Neural networks (NN) rely on the inner structure of available data sets rather than on comprehension of the modeled processes between inputs and outputs. Therefore, neural networks have been regarded as highly empirical models with limited extrapolation capability to situations outside the range of the training and validation data sets. In this study, the generalization ability of neural networks in predicting rice tillering dynamics was tested and several techniques inducing the generalization ability of neural networks were compared. We compared the performance of cross-validated neural networks with independent-validated neural networks and found that neural networks were able to extrapolate and predict tillering dynamics if the data were within the range of inputs of the training set. An inadequate training set resulted in overfitting of available data and neural networks that were not generalizable. The training set size required to enable a neural network to generalize and predict rice tillering dynamics was found to be at least 9 times as many training patterns for each weight. When a large number of variables are included in the input vector of a neural network with inadequate amounts of training data, we strongly recommend that the dimension of the input vector is reduced using principle component analysis (PCA), correspondence analysis (CA) or similar techniques to decrease the number of weights before the training procedure to improve the generalization ability of the NN. If the amount of training data still is not sufficient after the dimension of the input vector is reduced, regularization techniques, such as early stopping, jittering, and especially the embedment of estimated results by a theoretical model into the training set, should be used to improve the generalization ability of the neural network. The generalization of neural networks presents a wide spectrum of problems, and the proposed approaches are not confined strictly to modelling rice tillering dynamics but can be applied to other agricultural and ecological systems.

Extracting symbolic rules from trained neural network ensembles

Neural network ensemble can significantly improve the generalization ability of neural network based systems. However, its comprehensibility is even worse than that of a single neural network becau...

Enhancing the Generalization Ability of Neural Networks by Using Gram-Schmidt Orthogonalization Algorithm

Enhancing the Generalization Ability of Neural Networks by Using Gram-Schmidt Orthogonalization Algorithm

Improvement of generalization ability for identifying dynamical systems by using universal learning networks

This paper studies how the generalization ability of models of dynamical systems can be improved by taking advantage of the second order derivatives of the outputs with respect to the external inputs. The proposed method can be regarded as a direct implementation of the well-known regularization technique using the higher order derivatives of the Universal Learning Networks (ULNs). ULNs consist of a number of interconnected nodes where the nodes may have any continuously differentiable nonlinear functions in them and each pair of nodes can be connected by multiple branches with arbitrary time delays. A generalized learning algorithm has been derived for the ULNs, in which both the first order derivatives (gradients) and the higher order derivatives are incorporated. First, the method for computing the second order derivatives of ULNs is discussed. Then, a new method for implementing the regularization term is presented. Finally, simulation studies on identification of a nonlinear dynamical system with noises are carried out to demonstrate the effectiveness of the proposed method. Simulation results show that the proposed method can improve the generalization ability of neural networks significantly, especially in terms that (1) the robust network can be obtained even when the branches of trained ULNs are destructed, and (2) the obtained performance does not depend on the initial parameter values.

Quantitative analysis of generalization ability of neural network with prior-knowledge

The theoretical analysis of neural network generalization is presented in this paper with techniques based on the PAC learning and the VC dimension. Generalization indexes are proposed to measure the approximation correctness of trained neural networks for given target functions. Neural networks with prior knowledge are discussed on the basis of the generalization measurement, and are proved to possess better generalization ability compared with ordinary networks. They usually require considerably fewer training examples while keeping better generalization performance

Synthesis of Neural Networks and PID Control for Performance Improvement of Industrial Robots

In this article, an approach for improving the performance of industrial robots using multilayer feedforward neural networks is presented. The controller based on this approach consists of two main components: a PID control and a neural network. The function of the neural network is to complement the PID control for the specific purpose of improving the performance of the system over time. Analytical and experimental results concerning this synthesis of neural networks and PID control are presented. The analytical results assert that the performance of PID-controlled industrial robots can be improved through proper utilization of the learning and generalization ability of neural networks. The experimental results, obtained through actual implementation using a commercial industrial robot, demonstrate the effectiveness of such control synthesis for practical applications. The results of this work suggest that neural networks could be added to existing PID-controlled industrial robots for performance improvement.

A review of Bayesian neural networks with an application to near infrared spectroscopy

MacKay's (1992) Bayesian framework for backpropagation is a practical and powerful means to improve the generalization ability of neural networks. It is based on a Gaussian approximation to the posterior weight distribution. The framework is extended, reviewed, and demonstrated in a pedagogical way. The notation is simplified using the ordinary weight decay parameter, and a detailed and explicit procedure for adjusting several weight decay parameters is given. Bayesian backprop is applied in the prediction of fat content in minced meat from near infrared spectra. It outperforms "early stopping" as well as quadratic regression. The evidence of a committee of differently trained networks is computed, and the corresponding improved generalization is verified. The error bars on the predictions of the fat content are computed. There are three contributors: The random noise, the uncertainty in the weights, and the deviation among the committee members. The Bayesian framework is compared to Moody's GPE (1992). Finally, MacKay and Neal's automatic relevance determination, in which the weight decay parameters depend on the input number, is applied to the data with improved results.

Learning and generalization in a two-layer neural network: The role of the Vapnik-Chervonvenkis dimension.

Bounds for the generalization ability of neural networks based on Vapnik-Chervonenkis (VC) theory are compared with statistical mechanics results for the case of the parity machine. For fixed phase space dimension, the VC dimension can grows arbitrarily by increasing the number K of hidden units. Generalization is impossible up to a critical number of training examples that grows with the VC dimension. The asymptotic decrease of the generalization error ${\mathrm{\ensuremath{\varepsilon}}}_{\mathit{G}}$ comes out independent of K and the VC bounds strongly overestimate ${\mathrm{\ensuremath{\varepsilon}}}_{\mathit{G}}$. This shows that phase space dimension and VC dimension can play independent and different roles for the generalization process.

拡大ラグランジュ関数法を用いた満足化学習によるニューラルネットワークの汎化能力の獲得

拡大ラグランジュ関数法を用いた満足化学習によるニューラルネットワークの汎化能力の獲得

Systems that can learn from examples: Replica calculation of uniform convergence bounds for perceptrons.

The generalization abilities of neural networks for inferring a rule on the basis of examples can be characterized by the convergence of the learning error to the generalization error with increasing size of the training set. Using the replica technique, we calculate the maximum difference between training and generalization error for the ensemble of all perceptrons trained by a teacher perceptron and the maximal generalization error for the perceptrons that have a training error equal to zero. The results are compared with the rigorous bounds provided by the Vapnik-Chervonenkis theorem

Use of influence diagrams and neural networks in modeling semiconductor manufacturing processes

An adaptive learning architecture for modeling manufacturing processes involving several control variables is described. The use of this architecture to process modeling and recipe synthesis for deposition rate, stress, and film thickness in low-pressure chemical vapor deposition (LPCVD) of undoped polysilicon is discussed. In this architecture the model for a process is generated by combining the qualitative knowledge of human experts, captured in the form of influence diagrams, and the learning abilities of neural networks for extracting the quantitative knowledge that relates the parameters of a process. To evaluate the merits of this methodology, the accuracy of these new models is compared to that of more conventional models generated by the use of first principles and/or statistical regression analysis. The models generated by the integration of influence diagrams and neural networks are shown to have half the error or less, even though given only half as much information in creating the models. Furthermore, it is shown that, by employing the generalization ability of neural networks in the synthesis algorithm, new recipes can be produced for the process. Two such recipes are generated for the LPCVD process. One is a zero-stress polysilicon film recipe; the second is a uniform deposition rate recipe which is based on the use of a nonuniform temperature distribution during deposition. >