Training Multilayer Neural Networks Research Articles

Accelerated learning by active example selection.

Much previous work on training multilayer neural networks has attempted to speed up the backpropagation algorithm using more sophisticated weight modification rules, whereby all the given training examples are used in a random or predetermined sequence. In this paper we investigate an alternative approach in which the learning proceeds on an increasing number of selected training examples, starting with a small training set. We derive a measure of criticality of examples and present an incremental learning algorithm that uses this measure to select a critical subset of given examples for solving the particular task. Our experimental results suggest that the method can significantly improve training speed and generalization performance in many real applications of neural networks. This method can be used in conjunction with other variations of gradient descent algorithms.

Neural network-based classification system for texture images with its applications

A new approach to interconnecting multilayer feedforward neural networks for tackling the problems of texture classification is proposed. The resulting classification system classifies textures via two stages; one to compress original co-occurrence feature patterns of high dimensionality to lower dimensional principal feature patterns, and the other to perform actual classification of textures using the principal features. Each stage is efficiently implemented by a trained multilayer feedforward neural network. Such a cascaded use of neural networks significantly reduces the computational complexity that is otherwise encountered in classifying large-scale texture images. Two practical applications of the system are provided, showing the direct applicability of the approach for real problem-solving.

A novel multilayer neural networks training algorithm that minimizes the probability of classification error

A multilayer neural networks training algorithm that minimizes the probability of classification error is proposed. The claim is made that such an algorithm possesses some clear advantages over the standard backpropagation (BP) algorithm. The convergence analysis of the proposed procedure is performed and convergence of the sequence of criterion realizations with probability of one is proven. An experimental comparison with the BP algorithm on three artificial pattern recognition problems is given.

Efficient learning algorithms for neural networks (ELEANNE)

This paper presents the development of several efficient learning algorithms for neural networks (ELEANNE). The ELEANNE 1 and ELEANNE 2 are two recursive least-squares learning algorithms, proposed for training single-layered neural networks with analog output. This paper also proposes a new optimization strategy for training single-layered neural networks, which provides the basis for the development of a variety of efficient learning algorithms. This optimization strategy is the source of the ELEANNE 3, a second-order learning algorithm for training single-layered neural networks with binary output. A simplified version of this algorithm, called ELEANNE 4, is also derived on the basis of some simplifying but reasonable assumptions. The two algorithms developed for single-layered neural networks provide the basis for the derivation of ELEANNE 5 and ELEANNE 6, which are proposed for training multilayered neural networks with binary output. The ELEANNE 7 is an efficient algorithm developed for training multilayered neural networks with either binary or analog output. >

INTERNAL ORGANIZATION OF CLASSIFIER NETWORKS TRAINED BY BACKPROPAGATION

Backpropagation is one of the most widely used methods for training multilayer neural networks, yet questions still exist regarding how the networks organize internally during (raining to represent the external training environment. This paper presents empirical measurements showing that feedforward networks, when trained on many separable and non-separable problems, learn a characteristic internal representation, herein called the Network Linear Transform (NLT), that is independent of: (a) the initial weights and cell biases, and (b) the number of hidden units. The internal decomposition (defined as the values of cell weights and biases) of the trained nets, however, is greatly dependent upon these quantities. For the case of orthogonal input patterns, the NLT captures a literal image of the training environment, while for linearly-independent and separable linearly-dependent training sets, the NLT: (a) captures characteristic features of the correct input patterns, (b) captures inverted versions of characteristic features of incorrect patterns, and (c) rejects features common to all pattern classes. For non-separable problems, the NLT captures statistical ensemble information about patterns in each training class. The hidden units act as difference detectors and thus convey information that distinguishes input patterns from one another. They separate the patterns into groups that are easily discriminated by the output cells. A linearized mathematical network model is developed that accurately reproduces weight matrices and cell responses for certain separable learning situations, and which supports the experimental findings given above.

Feature-based texture classification of side-scan sonar images using a neural network approach

A texture classification system for side-scan sonar images by using a trained multilayer feedforward neural network (MFNN) is presented. The system classified textures by exploiting principal feature patterns, giving a high correct-classification rate. Experimental examples for the classification of side-scan sonar images are provided.

New learning algorithm for training multilayered neural networks that uses genetic-algorithm techniques

The Letter discusses a new method for training multilayered neural networks that uses genetic-algorithm techniques. Tests show that this method, called GenLearn, is significantly faster than methods that use the generalised delta rule. Gen-Learn also has the advantage that, unlike the generalised delta rule, it can escape local minima in its search of weight space.

A simple method to derive bounds on the size and to train multilayer neural networks

A new derivation is presented for the bounds on the size of a multilayer neural network to exactly implement an arbitrary training set; namely the training set can be implemented with zero error with two layers and with the number of the hidden-layer neurons equal to #1>/= p-1. The derivation does not require the separation of the input space by particular hyperplanes, as in previous derivations. The weights for the hidden layer can be chosen almost arbitrarily, and the weights for the output layer can be found by solving #1+1 linear equations. The method presented exactly solves (M), the multilayer neural network training problem, for any arbitrary training set.

Inversion of neural networks by gradient descent

Inversion answers the question of which input patterns to a trained multilayer neural network approximate a given output target. This method is a tool for visualization of the information processing capability of a network stored in its weights. This knowledge about the network enables one to make informed decisions on the improvement of the training task and the choice of training sets. An inversion algorithm for multilayer perceptrons is derived from the backpropagation scheme. We apply inversion to networks for digit recognition. We observe that the multilayer perceptrons perform well with respect to generalization, i.e. correct classification of untrained digits. They are however bad on rejection of counterexamples, i.e. random patterns. Inversion gives an explanation for this drawback. We suggest an improved training scheme, and we show that a tradeoff exists between generalization and rejection of counterexamples.

Training multi-layered neural networks with T.R. Based algorithm

Training multi-layered neural networks with T.R. Based algorithm