Design Of Feedforward Neural Networks Research Articles

Design of Feedforward Neural Networks in the Classification of Hyperspectral Imagery Using Superstructural Optimization

Artificial Neural Networks (ANNs) have been used in a wide range of applications for complex datasets with their flexible mathematical architecture. The flexibility is favored by the introduction of a higher number of connections and variables, in general. However, over-parameterization of the ANN equations and the existence of redundant input variables usually result in poor test performance. This paper proposes a superstructure-based mixed-integer nonlinear programming method for optimal structural design including neuron number selection, pruning, and input selection for multilayer perceptron (MLP) ANNs. In addition, this method uses statistical measures such as the parameter covariance matrix in order to increase the test performance while permitting reduced training performance. The suggested approach was implemented on two public hyperspectral datasets (with 10% and 50% sampling ratios), namely Indian Pines and Pavia University, for the classification problem. The test results revealed promising performances compared to the standard fully connected neural networks in terms of the estimated overall and individual class accuracies. With the application of the proposed superstructural optimization, fully connected networks were pruned by over 60% in terms of the total number of connections, resulting in an increase of 4% for the 10% sampling ratio and a 1% decrease for the 50% sampling ratio. Moreover, over 20% of the spectral bands in the Indian Pines data and 30% in the Pavia University data were found statistically insignificant, and they were thus removed from the MLP networks. As a result, the proposed method was found effective in optimizing the architectural design with high generalization capabilities, particularly for fewer numbers of samples. The analysis of the eliminated spectral bands revealed that the proposed algorithm mostly removed the bands adjacent to the pre-eliminated noisy bands and highly correlated bands carrying similar information.

Metaheuristic design of feedforward neural networks: A review of two decades of research

Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era.

Gaussian kernel approximation algorithm for feedforward neural network design

A Gaussian kernel approximation algorithm for a feedforward neural network is presented. The approach used by the algorithm, which is based on a constructive learning algorithm, is to create the hidden units directly so that automatic design of the architecture of neural networks can be carried out. The algorithm is defined using the linear summation of input patterns and their randomized input weights. Hidden-layer nodes are defined so as to partition the input space into homogeneous regions, where each region contains patterns belonging to the same class. The largest region is used to define the center of the corresponding Gaussian hidden nodes. The algorithm is tested on three benchmark data sets of different dimensionality and sample sizes to compare the approach presented here with other algorithms. Real medical diagnoses and a biological classification of mushrooms are used to illustrate the performance of the algorithm. These results confirm the effectiveness of the proposed algorithm.

Fast unit pruning algorithm for feedforward neural network design

A fast unit pruning algorithm for feedforward neural network is presented and the way used by the algorithm which based on optimal brain surgeon (OBS) is to remove the unneeded hidden units directly so that carry out the self-organization design on the architecture of neural networks. The algorithm is tested on several modeling problems, and is compared with OBS. It is found that the fast unit pruning algorithm is much more efficient than OBS which can not only reduce the complexity of the network but also accelerate the learning speed.

Geometric preprocessing and neurocomputing for pattern recognition and pose estimation

This paper shows the analysis and design of feed-forward neural networks using the coordinate-free system of Clifford or geometric algebra. It is shown that real-, complex- and quaternion-valued neural networks are simply particular cases of the geometric algebra multidimensional neural networks and that they can be generated using Support Multi-Vector Machines. Particularly, the generation of RBF for neurocomputing in geometric algebra is easier using the SMVM that allows to find the optimal parameters automatically. The use of SVM in the geometric algebra framework expands its sphere of applicability for multidimensional learning. We introduce a novel method of geometric preprocessing utilizing hypercomplex or Clifford moments. This method is applied together with geometric MLPs for tasks of 2D pattern recognition. Interesting examples of non-linear problems like the grasping of an object along a non-linear curve and the 3D pose recognition show the effect of the use of adequate Clifford or geometric algebras that alleviate the training of neural networks and that of Support Multi-Vector Machines.

Multiplierless multilayer feedforward neural network design using quantised neurons

An approach for designing multiplierless multilayer feedforward neural networks using quantised neurons is proposed. The design has significant merits for digital VLSI implementation in terms of reduced silicon area and increased operational speed. As compared to one-powers-of-two weights approaches of multiplierless design, the quantised neurons approach offers additional advantages of simplicity in activation function realisation and freedom in weights adaptation.

Geometric neural computing

This paper shows the analysis and design of feedforward neural networks using the coordinate-free system of Clifford or geometric algebra. It is shown that real-, complex-, and quaternion-valued neural networks are simply particular cases of the geometric algebra multidimensional neural networks and that some of them can also be generated using support multivector machines (SMVMs). Particularly, the generation of radial basis function for neurocomputing in geometric algebra is easier using the SMVM, which allows one to find automatically the optimal parameters. The use of support vector machines in the geometric algebra framework expands its sphere of applicability for multidimensional learning. Interesting examples of nonlinear problems show the effect of the use of an adequate Clifford geometric algebra which alleviate the training of neural networks and that of SMVMs.

Feedforward neural network design with tridiagonal symmetry constraints

This paper introduces a pruning algorithm with tridiagonal symmetry constraints for feedforward neural network (FANN) design. The algorithm uses a reflection transform applied to the input-hidden weight matrix in order to reduce it to its tridiagonal form. The designed FANN structures obtained by applying the proposed algorithm are compact and symmetrical. Therefore, they are well suited for efficient hardware and software implementations. Moreover, the number of the FANN parameters is reduced without a significant loss in performance. We illustrate the complexity and performance of the proposed algorithm by applying it as a solution to a nonlinear regression problem. We also compare the results of our proposed algorithm with those of the optimal brain damage algorithm.

Orthogonal considerations in the design of neural networks for function approximation

Abstract Two problems occur in the design of feedforward neural networks: the choice of the optimal architecture and the initialization. Generally, input and output data of a system (or a function) are measured and recorded. Then, experimenters wish to design a neural network to map exactly these output values. By formulating this as a continuous approximation problem, this paper shows that the use of orthogonal functions is a partial optimization in the choice of hidden functions. Parameter's initialization is obtained by using the knowledge of input and output data in the calculation of a discrete approximation. The hidden weights are found by constructing orthogonal directions on which the input values are represented. The pseudo-inverse is used to determine output weights such that the Euclidean distances between neural responses and output values are minimised.

Comments on “On the design of feedforward neural networks for binary mapping [1

In the paper [1], the authors present a new design technique that builds a feedforward net for an arbitrary set of binary associations. The design method is a decomposition-based design technique and claim that the average number of intermediate operation is around 1 2 nk for A and B of the same dimension ( k × n). We will propose an alternative method, based on the augmented operation and the perceptron mapping, to reduce the number of layers to two.

Network-growth approach to design of feedforward neural networks

A critical issue in applying the multilayer feedforward networks is the need to predetermine an appropriate network size for the problem being solved. A network-growth approach is pursued to address the problems concurrently and a progressive-training (PT) algorithm is proposed. The algorithm starts training with a one-hidden-node network and a one-pattern training subset. The training subset is then expanded by including one more pattern and the previously trained network, with or without a new hidden node grown, is trained again to cater for the new pattern. Such a process continues until all the available training patterns have been taken into account. At each training stage, convergence is guaranteed and at most one hidden node is added to the previously trained network. Thus the PT algorithm is guaranteed to converge to a finite-size network with a global minimum solution.

On the design of feedforward neural networks for binary mappings

This paper presents a new design technique that builds a feedforward net for an arbitrary set of binary associations. The key idea of the technique is to treat the set of associations as a binary matrix mapping, and to decompose the mapping into a cascade of the so-called primitive operations. There are only three different types of primitive operations, each of which is easily realizable into a simple feedforward subnet. The mapping, thus the set of associations, can then be readily realized by cascading the simple feedforward subnets that realize the primitive operations. The new technique has the important advantage that the computation time spent in constructing the realization is proportional to k · n, compared to k · 2 n typical of many conventional direct design approaches, where k is the number of given binary patterns, n the dimension of the patterns. Further, the generated feedforward net has k · n number of neurons, the same order of magnitude compared to many conventional methods. With such a substantial reduction in computational time, the technique can be readily employed in constructing feedforward nets for binary association problems of extremely large size.

A review of combinatorial problems arising in feedforward neural network design

This paper is primarily oriented towards discrete mathematics and emphasizes the occurrence of combinatorial problems in the area of artificial neural networks. The focus is on feedforward networks of binary units and their use as associative memories. Exact and heuristic algorithms for designing networks with single or multiple layers are discussed and complexity results related to the learning problems are reviewed. Several methods do only vary the parameters of networks whose topology has been chosen a priori while others build the networks during the training process. Valiant’s learning from examples model which formalizes the problem of generalization is presented and open questions are mentioned.

Multiplierless multilayer feedforward neural network design suitable for continuous input-output mapping

A method for designing a multiplierless multilayer feedforward neural network (MFNN) suitable for continuous input-output mapping is presented. When tested with noisy vectors, the network can retain a similar recall accuracy as the corresponding MFNN with continuous weights. The advantages of the design method include faster computational speed and reduced digital hardware cost.