Outputs Of Hidden Units Research Articles

Pruning of basis functions in nonlinear approximators

To obtain both good training performance and good generalization in multilayer perceptron (MLP) networks, it is essential to use small networks that avoid overfitting the training data. A common approach for doing this is to train a large network and then to prune the unnecessary units or weights. An effective hidden unit-pruning algorithm called linear dependence (LD) pruning utilizing sets of linear equations is presented in this paper. In this approach, hidden unit outputs (basis functions) are modeled as a linear combination of outputs of other units. The least useful hidden unit is identified as that which is predicted to increase training error the least when replaced by its model. After this hidden unit is found, the new pruning algorithm replaces it with its model and retrains the network output weights by one iteration of training. The LD pruning algorithm's performance is compared with that of a modified optimal brain surgeon (OBS) pruning algorithm. We show that the LD pruning algorithm performs as well as the OBS method, yet requires orders of magnitude fewer multiplies.

Temperature-Constrained Cascade Correlation Networks

A novel neural network has been devised that combines the advantages of cascade correlation and computational temperature constraints. The combination of advantages yields a nonlinear calibration method that is easier to use, stable, and faster than back-propagation networks. Cascade correlation networks adjust only a single unit at a time, so they train very rapidly when compared to back-propagation networks. Cascade correlation networks determine their topology during training. In addition, the hidden units are not readjusted once they have been trained, so these networks are capable of incremental learning and caching. With the cascade architecture, temperature may be optimized for each hidden unit. Computational temperature is a parameter that controls the fuzziness of a hidden unit's output. The magnitude of the change in covariance with respect to temperature is maximized. This criterion avoids local minima, forces the hidden units to model larger variances in the data, and generates hidden units that furnish fuzzy logic. As a result, models built using temperature-constrained cascade correlation networks are better at interpolation or generalization of the design points. These properties are demonstrated for exemplary linear interpolations, a nonlinear interpolation, and chemical data sets for which the numbers of chlorine atoms in polychlorinated biphenyl molecules are predicted from mass spectra.

Optimization of the hidden unit function in feedforward neural networks

A novel objective function is proposed for optimizing the hidden unit function in feedforward neural networks. This objective function represents the performance of the hidden unit at minimizing the least squared output errors of the linear output unit. This is derived from the decrease in the output errors due to the addition of the hidden units. The optimized output state vectors of the hidden units span a proper state space, which includes the desired output vectors for the network. The optimization (maximization of the objective function) is equal to minimizing the angle between the desired output vector and the projection of the hidden unit's output state vector onto the orthogonal complement of the subspace spanned by the other state vectors. The approximate solution can be obtained using the gradient ascent algorithm. This optimization method is useful in constructing fully connected feedforward neural networks and for minimizing the size of layered networks.