Generalization Capability Of Network Research Articles

A neural network approach for the simultaneous retrieval of volcanic ash parameters and SO<sub>2</sub> using MODIS data

Abstract. In this work neural networks (NNs) have been used for the retrieval of volcanic ash and sulfur dioxide (SO2) parameters based on Moderate Resolution Imaging Spectroradiometer (MODIS) multispectral measurements. Different neural networks were built in order for each parameter to be retrieved, for experimenting with different topologies and evaluating their performances. The neural networks' capabilities to process a large amount of new data in a very fast way have been exploited to propose a novel applicative scheme aimed at providing a complete characterization of eruptive products. As a test case, the May 2010 Eyjafjallajókull eruption has been considered. A set of seven MODIS images have been used for the training and validation phases. In order to estimate the parameters associated to the volcanic eruption, such as ash mass, effective radius, aerosol optical depth and SO2 columnar abundance, the neural networks have been trained using the retrievals from well-known algorithms. These are based on simulated radiances at the top of the atmosphere and are estimated by radiative transfer models. Three neural network topologies with a different number of inputs have been compared: (a) three thermal infrared MODIS channels, (b) all multispectral MODIS channels and (c) the channels selected by a pruning procedure applied to all MODIS channels. Results show that the neural network approach is able to estimate the volcanic eruption parameters very well, showing a root mean square error (RMSE) below the target data standard deviation (SD). The network built considering all the MODIS channels gives a better performance in terms of specialization, mainly on images close in time to the training ones, while the networks with less inputs reveal a better generalization performance when applied to independent data sets. In order to increase the network's generalization capability and to select the most significant MODIS channels, a pruning algorithm has been implemented. The pruning outcomes revealed that channel sensitive to ash parameters correspond to the thermal infrared, visible and mid-infrared spectral ranges. The neural network approach has been proven to be effective when addressing the inversion problem for the estimation of volcanic ash and SO2 cloud parameters, providing fast and reliable retrievals, important requirements during volcanic crises.

Improving the Generalization Ability of an Artificial Neural Network in Predicting In-Flight Particle Characteristics of an Atmospheric Plasma Spray Process

This paper presents the application of the artificial neural network into an atmospheric plasma spray process for predicting the in-flight particle characteristics, which have significant influence on the in-service coating properties. One of the major problems for such function-approximating neural network is over-fitting, which reduces the generalization capability of a trained network and its ability to work with sufficient accuracy under a new environment. Two methods are used to analyze the improvement in the network’s generalization ability: (i) cross-validation and early stopping, and (ii) Bayesian regularization. Simulations are performed both on the original and expanded database with different training conditions to obtain the variations in performance of the trained networks under various environments. The study further illustrates the design and optimization procedures and analyzes the predicted values, with respect to the experimental ones, to evaluate the performance and generalization ability of the network. The simulation results show that the performance of the trained networks with regularization is improved over that with cross-validation and early stopping and, furthermore, the generalization capability of the networks is improved; thus preventing any phenomenon associated with over-fitting.

OPTIMUM NUMBERS OF SINGLE NETWORK FOR COMBINATION IN MULTIPLE NEURAL NETWORKS MODELING APPROACH FOR MODELING NONLINEAR SYSTEM

This paper is focused on finding the optimum number of single networks in multiple neural networks combination to improve neural network model robustness for nonlinear process modeling and control. In order to improve the generalization capability of single neural network based models, combining multiple neural networks is proposed in this paper. By studying the optimum number of network that can be combined in multiple network combination, the researcher can estimate the complexity of the proposed model then obtained the exact number of networks for combination. Simple averaging combination approach is implemented in this paper which is applied to nonlinear process models. It is shown that the optimum number of networks for combination can be obtained hence enhancing the performance of the proposed model.

A Study of Genetic Neural Network as Classifiers and its Application in Breast Cancer Diagnosis

In this paper, a genetic neural network classification model is developed. The proposed model can not only optimize the weights and thresholds of neural network, but also reduce network size by identifying the feature subset effectively using genetic algorithm. This technique is aimed at finding out a network that is large enough to capture the accurate class attributes of the data as much as possible, while retaining the generalization capability of neural network. Breast cancer datasets (BCD) in UCI Machine Learning Repository are utilized to evaluate the proposed genetic neural network approach in this paper. Simulation results show that the developed model achieved dimensional reduction and the identification of benign and malignant tumors. Accordingly, it improved classification accuracy and demonstrated excellent classification efficiency.

Application and Analysis of RBF Neural Network for Burr Prediction in Micro-Machining

Micro-milling is widely used in material removal processes in industry. However, burrs are often formed on workpiece edges in milling process. Burr effects the dimensional tolerance and performance of the workpiece seriously and is desirable to be controlled. Burrs prediction technology is useful for cutting conditions optimization to control burrs forming. Due to lots of factors influencing the formation process of burr, it is a difficult task to establish the burr size prediction model by mathematical and mechanical method. RBF neural network was used for burr formation predition. Design of the network, network structure parameters determination and generalization capability of the network were analyzed and discussed. Achieved network has good fitting performance and generalization capability validated by experiments.

The role of feature selection in artificial neural network applications

Determination of the 'best' bands that are assigned to the input neurons of an artificial neural network (ANN) is one of the critical steps in designing the ANN for a particular problem. A large number of inputs reduces the network's generalization capabilities and introduces redundant and perhaps irrelevant information, while a small number of inputs could be insufficient for the network to learn the characteristics of the training data. The number of input bands defines the complexity of the problem. Methods used to select the optimum inputs are known as feature selection techniques. Their use in the context of artificial neural networks was investigated in this study. Statistical separability measures, specifically Wilks' v and Hotelling's T 2, and separability indices were employed to determine the best eight-band combination for two multispectral, multitemporal and multisensor image datasets. The Mahalanobis distance classifier was employed in the determination of the 'best' subset solution. In the search for the 'best' band combinations, two widely used search procedures, sequential forward selection and the genetic algorithm, were applied.

Stochastic neural computation. II. Soft competitive learning

For pt. I see ibid., p.891-905. An investigation has been made into the use of stochastic arithmetic to implement an artificial neural network solution to a typical pattern recognition application. Optical character recognition is performed on very noisy characters in the E-13B MICR font. The artificial neural network is composed of two layers, the first layer being a set of soft competitive learning subnetworks and the second a set of fully connected linear output neurons. The observed number of clock cycles in the stochastic case represents an order of magnitude improvement over the floating-point implementation assuming clock frequency parity. Network generalization capabilities were also compared based on the network squared error as a function of the amount of noise added to the input patterns. The stochastic network maintains a squared error within 10 percent of that of the floating-point implementation for a wide range of noise levels.

Stochastic neural computation. I. Computational elements

This paper examines a number of stochastic computational elements employed in artificial neural networks, several of which are introduced for the first time, together with an analysis of their operation. We briefly include multiplication, squaring, addition, subtraction, and division circuits in both unipolar and bipolar formats, the principles of which are well-known, at least for unipolar signals. We have introduced several modifications to improve the speed of the division operation. The primary contribution of this paper, however, is in introducing several state machine-based computational elements for performing sigmoid nonlinearity mappings, linear gain, and exponentiation functions. We also describe an efficient method for the generation of, and conversion between, stochastic and deterministic binary signals. The validity of the present approach is demonstrated in a companion paper through a sample application, the recognition of noisy optical characters using soft competitive learning. Network generalization capabilities of the stochastic network maintain a squared error within 10 percent of that of a floating-point implementation for a wide range of noise levels. While the accuracy of stochastic computation may not compare favorably with more conventional binary radix-based computation, the low circuit area, power, and speed characteristics may, in certain situations, make them attractive for VLSI implementation of artificial neural networks.

Neuro-genetic approach to multidimensional fuzzy reasoning for pattern classification

To tackle the pattern classification problems first we give a new interpretation to the multidimensional fuzzy implication (MFI). This new interpretation of MFI is used for multidimensional fuzzy reasoning (MFR) for pattern classification. We realize the new interpretation through multilayer perceptron. The learning scheme of the network is based on genetic algorithm (GA). A weight smoothing scheme is also proposed to improve neural network's generalization capability. The smoothing constraint is incorporated into the objective function of the network to reflect the neighborhood correlation and to seek those solutions which have smooth connection weights. At the learning stage of the neural network fuzzy linguistic statements have been used. Once learned, the nonfuzzy features of a pattern can be classified using a fuzzy masking. The performance of the proposed scheme is tested through synthetic data. Finally, we apply the proposed scheme to the vowel recognition problem of one Indian language.

A Sequential Learning Approach for Single Hidden Layer Neural Networks

A sequential orthogonal approach to the building and training of single hidden layer neural networks is presented in this paper. In the proposed method, hidden neurons are added one at a time. The procedure starts with a single hidden neuron and sequentially increases the number of hidden neurons until the model error is sufficiently small. When adding a neuron, the new information introduced by this neuron is caused by that part of its output vector which is orthogonal to the space spanned by the output vectors of previously added hidden neurons. The classical Gram–Schmidt orthogonalization method is used at each step to form a set of orthogonal bases for the space spanned by the output vectors of hidden neurons. Hidden layer weights are found through optimization while output layer weights are obtained from the least-squares regression. Using the proposed technique, it is possible to determine the necessary number of hidden neurons required. A regularization factor is also incorporated into the sequential orthogonal training algorithm to improve the network generalization capability. An additional advantage of this method is that it can be used to build and train neural networks with mixed types of hidden neurons and thus to develop hybrid models. By using mixed types of neurons, it is found that more accurate neural network models, with a smaller number of hidden neurons than seen in conventional networks, can be developed. The proposed sequential orthogonal training method was successfully applied to three non-linear modelling examples.

A relaxation algorithm for estimating the domain of validity of feedforward neural networks

We expand on a recent paper by Courrieu which introduces three algorithms for determining the distance between any point and the interpolation domain associated with a feedforward neural network. This has been shown to have a significant relation with the network's generalization capability. A further neural-like relaxation algorithm is presented here, which is proven to naturally solve the problem originally posed by Courrieu. The algorithm is based on a powerful result developed in the context of Markov chain theory, and turns out to be a special case of a more general relaxation model which has long become a standard technique in the machine vision domain. Some experiments are presented which confirm the validity of the proposed approach.

Weight smoothing to improve network generalization

A weight smoothing algorithm is proposed in this paper to improve a neural network's generalization capability. The algorithm can be used when the data patterns to be classified are presented on an n-dimensional grid (n>/=1) and there exists some correlations among neighboring data points within a pattern. For a fully-interconnected feedforward net, no such correlation information is embedded into the architecture. Consequently, the correlations can only be extracted through sufficient amount of network training. With the proposed algorithm, a smoothing constraint is incorporated into the objective function of backpropagation to reflect the neighborhood correlations and to seek those solutions that have smooth connection weights. Experiments were performed on problems of waveform classification, multifont alphanumeric character recognition, and handwritten numeral recognition. The results indicate that (1) networks trained with the algorithm do have smooth connection weights, and (2) they generalize better.

Resolving multifont character confusion with neural networks

The existence of similar characters contributes to most of the errors made by multifont character recognition systems. A set of feedforward neural networks and a rule invocation procedure are added to a conventional character recognition system to help resolve similar character confusion. Compared to previous methods, the approach has the advantage of automating the feature selection and feature extraction processes, and it is more robust to noise as confirmed by experiments. The contributions are a snowball training algorithm and a smoothing technique, both are modifications to the back-propagation training algorithm. The snowball training algorithm presents training data in a better sequence to remedy the training convergence problem. The smoothing technique seeks a solution with smooth connection weights during training to improve a network's generalization capability. Combining the two proposed techniques increases the average recognition rate of the rule nets to 99.65 from 87.35%.