Feedforward Neural Network Classifier Research Articles

Analysis of Tikhonov regularization for function approximation by neural networks

This paper is devoted to the convergence and stability analysis of Tikhonov regularization for function approximation by a class of feed-forward neural networks with one hidden layer and linear output layer. We investigate two frequently used approaches, namely regularization by output smoothing and regularization by weight decay, as well as a combination of both methods to combine their advantages. We show that in all cases stable approximations are obtained converging to the approximated function in a desired Sobolev space as the noise in the data tends to zero (in the weaker L 2-norm) if the regularization parameter and the number of units in the network are chosen appropriately. Under additional smoothness assumptions we are able to show convergence rates results in terms of the noise level and the number of units in the network. In addition, we show how the theoretical results can be applied to the important classes of perceptrons with one hidden layer and to translation networks. Finally, the performance of the different approaches is compared in some numerical examples.

Neural network based system for script identification in Indian documents

The paper describes a neural network-based script identification system which can be used in the machine reading of documents written in English, Hindi and Kannada language scripts. Script identification is a basic requirement in automation of document processing, in multi-script, multi-lingual environments. The system developed includes a feature extractor and a modular neural network. The feature extractor consists of two stages. In the first stage the document image is dilated using 3 X 3 masks in horizontal, vertical, right diagonal, and left diagonal directions. In the next stage, average pixel distribution is found in these resulting images. The modular network is a combination of separately trained feedforward neural network classifiers for each script. The system recognizes 64 X 64 pixel document images. In the next level, the system is modified to perform on single word-document images in the same three scripts. Modified system includes a pre-processor, modified feature extractor and probabilistic neural network classifier. Pre-processor segments the multi-script multi-lingual document into individual words. The feature extractor receives these word-document images of variable size and still produces the discriminative features employed by the probabilistic neural classifier. Experiments are conducted on a manually developed database of document images of size 64 X 64 pixels and on a database of individual words in the three scripts. The results are very encouraging and prove the effectiveness of the approach.

A feedforward multi-layer neural network for machine cell formation in computer integrated manufacturing

Neural-network applications have been one of the better alternatives either for simulating massive data in parallel, or embedding human subjective decisions into existing quantitative models, thereby spawning a qualitative model. This paper introduces a linear classifier with a classical feedforward neural network in forming machine cells or groups for Computer Integrated Manufacturing. The proposed method, through experiment, has been proven to outperform conventional methods such as Part Family Analysis (PFA) and BLOCPLAN, among others. A single-layer perceptron, along with multi-layer feedforward network where applicable, have been employed in forming the part families. The underlying philosophy is the Group Technology (GT). The developed models and algorithms are illustrated with a numerical example.

SEPARATING SEPTICEMIC AND NORMAL CHICKEN LIVERS BY VISIBLE/NEAR–INFRARED SPECTROSCOPY AND BACK–PROPAGATION NEURAL NETWORKS

The visible/nearinfrared spectra of 300 chicken livers were analyzed to explore the feasibility of usingspectroscopy to separate septicemic livers from normal livers. Three strategies involving offset, second difference, andfunctional link methods were applied to preprocess the spectra, while principal component analysis (PCA) was utilized toreduce the input data dimensions. PCA scores were fed into a feedforward backpropagation neural network forclassification. The results showed no obvious difference in classification accuracy between offset and nonoffset data whenno other preprocessing method was applied. The full 4002498 nm wavelength region produced better results than the400700 nm, 4001098 nm, and 11022498 nm subregions when more than 30 PCA scores were used. In general, theclassification accuracy was improved by increasing the number of scores of input data, but too many scores diminishedperformance. The functional link test showed that using functionallink spectra selected at every third point with 60 scoresachieved the same classification accuracy as that obtained when using all the data points with 90 scores. The bestclassification model used offset correction followed by second difference (g = 31) and 60 scores. It achieved a classificationaccuracy of 98% for normal and 94% for septicemic livers.

A class-modular feedforward neural network for handwriting recognition

Since the conventional feedforward neural networks for character recognition have been designed to classify a large number of classes with one large network structure, inevitably it poses the very complex problem of determining the optimal decision boundaries for all the classes involved in a high-dimensional feature space. Limitations also exist in several aspects of the training and recognition processes. This paper introduces the class modularity concept to the feedforward neural network classifier to overcome such limitations. In the class-modular concept, the original K-classification problem is decomposed into K 2-classification subproblems. A modular architecture is adopted which consists of K subnetworks, each responsible for discriminating a class from the other K−1 classes. The primary purpose of this paper is to prove the effectiveness of class-modular neural networks in terms of their convergence and recognition power. Several cases have been studied, including the recognition of handwritten numerals (10 classes), English capital letters (26 classes), touching numeral pairs (100 classes), and Korean characters in postal addresses (352 classes). The test results confirmed the superiority of the class-modular neural network and the interesting aspects on further investigations of the class modularity paradigm.

A use of a neural network to evaluate contrast enhancement curves in breast magnetic resonance images.

For the diagnosis of breast cancer using magnetic resonance imaging (MRI), one of the most important parameters is the analysis of contrast enhancement. A three-dimensional MR sequence is applied before and five times after bolus injection of paramagnetic contrast medium (Gd-DTPA). The dynamics of absorption are described by a time/intensity enhancement curve, which reports the mean intensity of the MR signal in a small region of interest (ROI) for about 8 minutes after contrast injection. The aim of our study was to use an artificial neural network to automatically classify the enhancement curves as "benign" or "malignant." We used a classic feed-forward back-propagation neural network, with three layers: five input nodes, two hidden nodes, and one output node. The network has been trained with 26 pathologic curves (10 invasive carcinoma [K], two carcinoma-in-situ [DCIS], and 14 benign lesion [B]). The trained network has been tested with 58 curves (36 K, one DCIS, 21 B). The network was able to correctly identify the test curves with a sensitivity of 76% and a specificity of 90%. For comparison, the same set of curves was analyzed separately by two radiologists (a breast MR expert and a resident radiologist). The first correctly interpreted the curves with a sensitivity of 76% and a specificity of 90%, while the second scored 59% for sensitivity and 90% for specificity. These results demonstrate that a trained neural network recognizes the pathologic curves at least as well as an expert radiologist. This algorithm can help the radiologist attain rapid and affordable screening of a large number of ROIs. A complete automatic computer-aided diagnosis support system should find a number of potentially interesting ROIs and automatically analyze the enhancement curves for each ROI by neural networks, reporting to the radiologist only the potentially pathologic ROIs for a more accurate, manual, repeated evaluation.

Estimating Labor Productivity Using Probability Inference Neural Network

This paper discusses the derivation of a probabilistic neural network classification model and its application in the construction industry. The probability inference neural network (PINN) model is based on the same concepts as those of the learning vector quantization method combined with a probabilistic approach. The classification and prediction networks are combined in an integrated network, which required the development of a different training and recall algorithm. The topology and algorithm of the developed model was presented and explained in detail. Portable computer software was developed to implement the training, testing, and recall for PINN. The PINN was tested on real historical productivity data at a local construction company and compared to the classic feedforward back-propagation neural network model. This showed marked improvement in performance and accuracy. In addition, the effectiveness of PINN for estimating labor production rates in the context of the application domain was validat...

Approximate nonlinear output regulation based on the universal approximation theorem

The regulator equations arising from the nonlinear output regulation problem are a set of mixed partial and algebraic equations. Due to the nonlinear nature, it is difficult to obtain the exact solution of the regulator equations. This paper presents an approximation method for solving the regulator equations based on a class of feedforward neural networks. It is shown that a three-layer neural network can solve the regulator equations up to a prescribed arbitrarily small error, and this small error can be translated into a guaranteed steady-state tracking error for the closed-loop system. The method has led to an effective approach to approximately solving the nonlinear output regulation problem. Copyright © 2000 John Wiley & Sons, Ltd.

PROFILING NEURAL NETWORKS FOR OPTION PRICING

In recent years the problem of option pricing has received increasing interest from both financial institutions and academics. It is well known that conventional modeling techniques for option pricing have inherent, persistent and systematic biases which are mainly due to the assumption of constant volatility for prices associated with the underlying financial instrument. Nowadays, there is strong and increasing evidence that financial markets are far from being stationary and then, whenever dealing with option pricing, we have to take into account the market heteroschedasticity. A possible approach for dealing with non-constant volatility relies on the modeling of the basic characteristics named implied volatility. Unfortunately this task is extremely complex and parametric models are not available. In this paper the authors discuss how models from the class of Feedforward Neural Networks can be exploited for approaching the task of implied volatility modeling. In particular the paper shows how the main techniques from the nonlinear regression framework can be exploited when models from the class of Feedforward Neural Networks are used. Indeed, in such a case the paucity of data, which can be used for the network training, and the particular structure of Feedforward Neural Networks make the modeling task numerically complex. The authors discuss how the nonlinear regression technique named profile can be exploited for selecting the optimal network's structure and evaluating its numeical properties. To this end, a numerical procedure for empirical model building, in the case of Feedforward Neural Networks, has been developed. Results are evaluated through an ad-hoc procedure which utilizes the estimated implied volatility surface for pricing general contingent claims. Numerical experiments, in the case of the USD/DEM options, are presented and discussed.

Mapping Land Cover from Remotely Sensed Data with a Softened Feedforward Neural Network Classification

Remote sensing has considerable potential as a source of data for land cover mapping. This potential remains to be fully realised due, in part, to the methods used to extract land cover information from the remotely sensed data. Widely used statistical classifiers provide a poor representation of land cover, make untenable assumptions about the data and convey no information on the quality of individual class allocations. This paper shows that a softened classification, providing information on the strength of membership to all classes for each image pixel, may be derived from a neural network. This information may be used to indicate classification quality on a per-pixel basis. Moreover, a soft or fuzzy classification may be derived to more appropriately represent land cover than the conventional hard classification.

The “bottleneck” behaviours in linear feedforward neural network classifiers and their breakthrough

The classification mechanisms of linear feedforward neural network classifiers (FNNC), whose hidden layer performs the Fisher linear transformation of the input patterns, under the supervision of outer-supervised signals are investigated. The “bottleneck” behaviours in linear FNNCs are observed and analyzed. In addition, the structure stabilities of the linear FNNCs are also discussed. It is pointed out that the key point to break through the “bottleneck” behaviours for linear FNNCs is to change linear hidden neurons into nonlinear hidden ones. Finally, the experimental results, taking the parity 3 problem as example, are given.

Linear and Nonlinear Feedforward Neural Network Classifiers: A Comprehensive Understanding

Linear and Nonlinear Feedforward Neural Network Classifiers: A Comprehensive Understanding

Neural network modeling in optimisation of continuous fermentation processes

The capability of self-recurrent neural networks in dynamic modeling of continuous fermentation is investigated in this simulation study. In the past, feedforward neural networks have been successfully used as one-step-ahead predictors. However, in steady-state optimisation of continuous fermentations the neural network model has to be iterated to predict many time steps ahead into the future in order to get steady-state values of the variables involved in objective cost function, and this iteration may result in increasing errors. Therefore, as an alternative to classical feedforward neural network trained by using backpropagation method, self-recurrent multilayer neural net trained by backpropagation through time method was chosen in order to improve accuracy of long-term predictions. Prediction capabilities of the resulting neural network model is tested by implementing this into the Integrated System Optimisation and Parameter Estimation (ISOPE) optimisation algorithm. Maximisation of cellular productivity of the baker's yeast continuous fermentation was used as the goal of the proposed optimising control problem. The training and prediction results of proposed neural network and performances of resulting optimisation structure are demonstrated.

A hybrid physical model—neural network approach to machine diagnostics

Neural network based diagnostic algorithms have been an effective approach to machine diagnostics. A feature extraction routine calculates parameters indicating the relative changes in signal phase along different paths to multiple accelerometers as might be due to localized damage along one path. These parameters were calculated for accelerometer data from the Westland seeded fault helicopter gearbox experiment. Perfect fault discrimination was achieved using a conventional feed forward neural network classifier on only a small training set, with no special treatment for the confounding influence of varying torque levels. A hybrid neural network was designed, which implemented a simple springs and masses mechanical model of a more complex structure, where sensor locations served as the mass points. This approach is able to localize structural damage using only a small training set on data generated by a finite elements model, and exhibits excellent interpolation and extrapolation. Extending the hybrid neural network approach, a method is proposed for solving for the external forces applied to surface-mounted sensors, from vibrating machine components. The derived forces will then serve as inputs to the signal phase change feature extraction routine. It is hoped this method will train well on small data sets and reduce ambiguities due to signal multipath effects. [See NOISE-CON Proceedings for full paper.]

Quantum neural networks (QNNs): inherently fuzzy feedforward neural networks

This paper introduces quantum neural networks (QNNs), a class of feedforward neural networks (FFNNs) inherently capable of estimating the structure of a feature space in the form of fuzzy sets. The hidden units of these networks develop quantized representations of the sample information provided by the training data set in various graded levels of certainty. Unlike other approaches attempting to merge fuzzy logic and neural networks, QNNs can be used in pattern classification problems without any restricting assumptions such as the availability of a priori knowledge or desired membership profile, convexity of classes, a limited number of classes, etc. Experimental results presented here show that QNNs are capable of recognizing structures in data, a property that conventional FFNNs with sigmoidal hidden units lack.

Grading Pistachio Nuts Using A Neural Network Approach

A multi-structure neural network (MSNN) classifier was proposed and applied to classify four varieties (classes) of pistachio nuts. The MSNN classifier consisted of four parallel discriminators (one per class), followed by a maximum selector. Each discriminator was a feed-forward neural network with two hidden layers and a single-neuron output layer. The discriminators were individually trained using physical attributes of the nuts extracted from their images as input. The performance of MSNN classifier was compared with the performance of a multi-layer feed-forward neural network (MLNN) classifier. The average classification accuracy of MSNN classifier was 95.9%, an increase of over 8.9% of the performance of MLNN.

The effect of weight precision and range on neural network classifier performance

This study investigates the precision and range requirements for weights in feed-forward neural network classifiers using backpropagation training with simulated signals where we could control the difficulty of the problem. We found that a precision of five bits and a range equal to four times the mean weight magnitude produced the same performance as continuous weights. However, we noted that the required precision and range depended strongly on the problem difficulty, the network complexity, and the relationship between these two factors. We also found that uniformly distributed discrete weights produced better performance than non-uniformly distributed ones.

Analysis and synthesis of feedforward neural networks using discrete affine wavelet transformations

A representation of a class of feedforward neural networks in terms of discrete affine wavelet transforms is developed. It is shown that by appropriate grouping of terms, feedforward neural networks with sigmoidal activation functions can be viewed as architectures which implement affine wavelet decompositions of mappings. It is shown that the wavelet transform formalism provides a mathematical framework within which it is possible to perform both analysis and synthesis of feedforward networks. For the purpose of analysis, the wavelet formulation characterizes a class of mappings which can be implemented by feedforward networks as well as reveals an exact implementation of a given mapping in this class. Spatio-spectral localization properties of wavelets can be exploited in synthesizing a feedforward network to perform a given approximation task. Two synthesis procedures based on spatio-spectral localization that reduce the training problem to one of convex optimization are outlined.

Heuristic configuration of single hidden-layer feed-forward neural networks

For optimum statistical classification and generalization with single hidden-layer neural network models, two tasks must be performed: (a) learning the best set of weights for a network of k hidden units and (b) determining k, the best complexity fit. We contrast two approaches to construction of neural network classifiers: (a) standard back-propagation as applied to a series of single hidden-layer feed-forward nerual networks with differing number of hidden units and (b) a heuristic cascade-correlation approach that quickly and dynamically configures the hidden units in a network and learns the best set of weights for it. Four real-world applications are considered. On these examples, the back-propagation approach yielded somewhat better results, but with far greater computation times. The best complexity fit, k, for both approaches were quite similar. This suggests a hybrid approach to constructing single hidden-layer feed-forward neural network classifiers in which the number of hidden units is determined by cascade-correlation and the weights are learned by back-propagation.

Admissible stochastic complexity models for classification problems

In this paper we investigate the application of stochastic complexity theory to classification problems. In particular, we define the notion of admissible models as a function of problem complexity, the number of data pointsN, and prior belief. This allows us to derive general bounds relating classifier complexity with data-dependent parameters such as sample size, class entropy and the optimal Bayes error rate. We discuss the application of these results to a variety of problems, including decision tree classifiers, Markov models for image segmentation, and feedforward multilayer neural network classifiers.