Single Hidden Layer Neural Network Research Articles

Efficient and effective algorithms for training single-hidden-layer neural networks

Recently there have been renewed interests in single-hidden-layer neural networks (SHLNNs). This is due to its powerful modeling ability as well as the existence of some efficient learning algorithms. A prominent example of such algorithms is extreme learning machine (ELM), which assigns random values to the lower-layer weights. While ELM can be trained efficiently, it requires many more hidden units than is typically needed by the conventional neural networks to achieve matched classification accuracy. The use of a large number of hidden units translates to significantly increased test time, which is more valuable than training time in practice. In this paper, we propose a series of new efficient learning algorithms for SHLNNs. Our algorithms exploit both the structure of SHLNNs and the gradient information over all training epochs, and update the weights in the direction along which the overall square error is reduced the most. Experiments on the MNIST handwritten digit recognition task and the MAGIC gamma telescope dataset show that the algorithms proposed in this paper obtain significantly better classification accuracy than ELM when the same number of hidden units is used. For obtaining the same classification accuracy, our best algorithm requires only 1/16 of the model size and thus approximately 1/16 of test time compared with ELM. This huge advantage is gained at the expense of 5 times or less the training cost incurred by the ELM training.

A Constructive Algorithm with Adaptive Sigmoidal Function for Designing Single Hidden Layer Feedforward Neural Network

In this paper we propose a constructive algorithm with adaptive sigmoidal function for designing single hidden layer feedforward neural network (CAASF). The proposed algorithm emphasizes on architectural adaptation and functional adaptation during training. This algorithm is a constructive approach to building single hidden layer neural networks dynamically. The activation functions used at non-linear hidden nodes are belonging to the well-defined sigmoidal class and adapted during training. The algorithm determines not only optimum number of hidden nodes, as also optimum sigmoidal function for the non-linear nodes. One simple variant derived from CAASF is where the sigmoidal function used at the hidden nodes is fixed. Both the variants are compared to each other on five regression functions. Simulation results reveal that adaptive sigmoidal function presents several advantages over traditional fixed sigmoid function, resulting in increased flexibility, smoother learning, better convergence and better generalization performance.

Direct adaptive neural network controller for a class of nonlinear systems based on fuzzy estimator of the control error

A new approach of direct adaptive control of single input single output nonlinear systems in affine form using single-hidden layer neural network (NN) is introduced. In contrast to the algorithms in the literature, the weights adaptation laws are based on the control error and not on the tracking error or its filtered version. Since the control error is being expressed in terms of the NN controller, hence its weights updating laws are obtained via back-propagation concept. A fuzzy inference system (FIS) with heuristically defined rules is introduced to provide an estimate of this error based on the past history of the system behaviour. The stability of the closed loop is studied using Lyapunov theory. A fixed structure is then proposed for the FIS and the design parameters reduce to the parameters of the NN. The method is reproducible and does not require any pre-training of the network weights.

Study of dead-end microfiltration features in sequencing batch reactor (SBR) by optimized neural networks

In this research, the structure optimized BP-ANN model was applied to simulate the permeate flux as a function of mixed liquor suspended, temperature, dissolved oxygen, hydraulic retention time, transmembrane pressure and operating time during dead-end microfiltration of activated sludge suspensions and its supernatant from sequencing batch reactor (SBR). This artificial neural network approach was also used to model the chemical oxygen demand (COD) concentration of effluent from SBR. The results showed that the structure optimized single hidden layer neural networks was able to accurately simulate the dynamic behavior of permeate flux and the COD concentration for SBR activated sludge process, and this BP-ANN model possessed higher accuracy than that of C. M. Silva's predictive model and linear multi-regression model.

An Iterative Method for Deciding SVM and Single Layer Neural Network Structures

We present two new classifiers for two-class classification problems using a new Beta-SVM kernel transformation and an iterative algorithm to concurrently select the support vectors for a support vector machine (SVM) and the hidden units for a single hidden layer neural network to achieve a better generalization performance. To construct the classifiers, the contributing data points are chosen on the basis of a thresholding scheme of the outputs of a single perceptron trained using all training data samples. The chosen support vectors are used to construct a new SVM classifier that we call Beta-SVN. The number of chosen support vectors is used to determine the structure of the hidden layer in a single hidden layer neural network that we call Beta-NN. The Beta-SVN and Beta-NN structures produced by our method outperformed other commonly used classifiers when tested on a 2-dimensional non-linearly separable data set.

Performance enhancement of extreme learning machine for multi-category sparse data classification problems

This paper presents a performance enhancement scheme for the recently developed extreme learning machine (ELM) for multi-category sparse data classification problems. ELM is a single hidden layer neural network with good generalization capabilities and extremely fast learning capacity. In ELM, the input weights are randomly chosen and the output weights are analytically calculated. The generalization performance of the ELM algorithm for sparse data classification problem depends critically on three free parameters. They are, the number of hidden neurons, the input weights and the bias values which need to be optimally chosen. Selection of these parameters for the best performance of ELM involves a complex optimization problem. In this paper, we present a new, real-coded genetic algorithm approach called ‘RCGA-ELM’ to select the optimal number of hidden neurons, input weights and bias values which results in better performance. Two new genetic operators called ‘network based operator’ and ‘weight based operator’ are proposed to find a compact network with higher generalization performance. We also present an alternate and less computationally intensive approach called ‘sparse-ELM’. Sparse-ELM searches for the best parameters of ELM using K-fold validation. A multi-class human cancer classification problem using micro-array gene expression data (which is sparse), is used for evaluating the performance of the two schemes. Results indicate that the proposed RCGA-ELM and sparse-ELM significantly improve ELM performance for sparse multi-category classification problems.

Control of uncertain systems by feedback linearization with neural networks augmentation. Part I. Controller design

The paper highlights the main steps of adaptive output feedback control for non-affine uncertain systems – both in parameters and dynamics – having a known relative degree. Given a reference model, the objective is to design a controller that forces the measured system output to track the reference model output with bounded errors. A single hidden layer neural network is used to counteract feedback linearization error. A dynamic observer of tracking error is added. The treatment of control saturation is also sketched. The mathematical model for the longitudinal dynamics of an experimental helicopter is used as framework.

Neighborhood based modified backpropagation algorithm using adaptive learning parameters for training feedforward neural networks

The major drawbacks of backpropagation algorithm are local minima and slow convergence. This paper presents an efficient technique ANMBP for training single hidden layer neural network to improve convergence speed and to escape from local minima. The algorithm is based on modified backpropagation algorithm in neighborhood based neural network by replacing fixed learning parameters with adaptive learning parameters. The developed learning algorithm is applied to several problems. In all the problems, the proposed algorithm outperform well.

Comparison of RBF and SHL Neural Network Based Adaptive Control

Modern unmanned aerial vehicles (UAVs) are required to perform complex maneuvers while operating in increasingly uncertain environments. To meet these demands and model the system dynamics with a high degree of precision, a control system design known as neural network based model reference adaptive control (MRAC) is employed. There are currently two neural network architectures used by industry and academia as the adaptive element for MRAC; the radial basis function and single hidden layer neural network. While mathematical derivations can identify differences between the two neural networks, there have been no comparative analyses conducted on the performance characteristics for the flight controller to justify the selection of one neural network over the other. While the architecture of both neural networks contain similarities, there are several key distinctions which exhibit a noticeable impact on the control system's overall performance. In this paper, a detailed comparison of the performance characteristics between both neural network based adaptive control approaches has been conducted in an application highly relevant to UAVs. The results and conclusions drawn from this paper will provide engineers with tangible justification for the selection of the better neural network adaptive element and thus a controller with better performance characteristics.

Stable direct adaptive neural network controller with a fuzzy estimator of the control error for a class of perturbed nonlinear systems

A state feedback direct adaptive control algorithm for single input single output perturbed nonlinear systems in affine form using single hidden layer neural network is introduced. The weights adaptation laws are based on an estimated control error provided by a fuzzy inference system composed of heuristically determined rules. It provides a bounded estimate of the control error, which affects only the step size of the updating laws. It is shown that under mild conditions the state variables and the control input are bounded and the tracking error and its derivatives converge to a bounded compact set. The method does not require any preliminary off line training of the network weights. All states are supposed to be measurable. Two simulation studies are presented for testing the proposed algorithm.

Geometrical synthesis of MLP neural networks

This paper presents a new constructive algorithm to design multilayer perceptron networks used as classifiers. The resulting networks are able to classify patterns defined in a real domain. The proposed procedure allows us to automatically determine both the number of neurons and the synaptic weights of networks with a single hidden layer. The approach is based on linear programming. It avoids the typical local minima problems of error back propagation and assures convergence of the method. Furthermore, it will be shown in this paper that the presented procedure leads to single-hidden layer neural networks able to solve any problem in classifying a finite number of patterns. The performances of the proposed algorithm have been tested on some benchmark problems, and they have been compared with those of different approaches.

Single hidden layer artificial neural network models versus multiple linear regression model in forecasting the time series of total ozone

Present paper endeavors to develop predictive artificial neural network model for forecasting the mean monthly total ozone concentration over Arosa, Switzerland. Single hidden layer neural network models with variable number of nodes have been developed and their performances have been evaluated using the method of least squares and error estimation. Their performances have been compared with multiple linear regression model. Ultimately, single-hidden-layer model with 8 hidden nodes have been identified as the best predictive model.

Constructing Low-Order Discriminant Neural Networks Using Statistical Feature Selection

CONSTRUCTING LOW-ORDER DISCRIMINANT NEURAL NETWORKS USING STATISTICAL FEATURE SELECTION The selection of relevant inputs, and determining an appropriate network topology, are two critical issues faced when applying neural networks to classification problems. This paper presents an algorithm called Pair Attribute Learning (PAL) for addressing both input selection, and the determination of network topology. The PAL algorithm uses a preprocessing stage to search for features derived from pairs of training instances. A statistical rank is used to select a good set of features, and these features are then used to drive the construction of a single hidden layer neural network. Only inputs relevant within the context of a feature are used in constructing the network. This results in a sparsely connected hidden layer, and lower-order discriminants. Results on nine learning problems demonstrate that PAL constructed networks are 70% less complex on average than networks built using other constructive techniques, without a significant loss of predictive accuracy. In addition, the PAL algorithm does not use iterative construction, or suffer from bias mismatch. Because it addresses both input selection and network topology, it provides an end-to-end solution for applying neural networks to classification problems.

Neural network prediction of carbonate lithofacies from well logs, Big Bow and Sand Arroyo Creek fields, Southwest Kansas

In the Hugoton Embayment of southwestern Kansas, St. Louis Limestone reservoirs have relatively low recovery efficiencies, attributed to the heterogeneous nature of the oolitic deposits. This study establishes quantitative relationships between digital well logs and core description data, and applies these relationships in a probabilistic sense to predict lithofacies in 90 uncored wells across the Big Bow and Sand Arroyo Creek fields. In 10 wells, a single hidden-layer neural network based on digital well logs and core described lithofacies of the limestone depositional texture was used to train and establish a non-linear relationship between lithofacies assignments from detailed core descriptions and selected log curves. Neural network models were optimized by selecting six predictor variables and automated cross-validation with neural network parameters and then used to predict lithofacies on the whole data set of the 2023 half-foot intervals from the 10 cored wells with the selected network size of 35 and a damping parameter of 0.01. Predicted lithofacies results compared to actual lithofacies displays absolute accuracies of 70.37–90.82%. Incorporating adjoining lithofacies, within-one lithofacies improves accuracy slightly (93.72%). Digital logs from uncored wells were batch processed to predict lithofacies and probabilities related to each lithofacies at half-foot resolution corresponding to log units. The results were used to construct interpolated cross-sections and useful depositional patterns of St. Louis lithofacies were illustrated, e.g., the concentration of oolitic deposits (including lithofacies 5 and 6) along local highs and the relative dominance of quartz-rich carbonate grainstone (lithofacies 1) in the zones A and B of the St. Louis Limestone. Neural network techniques are applicable to other complex reservoirs, in which facies geometry and distribution are the key factors controlling heterogeneity and distribution of rock properties. Future work involves extension of the neural network to predict reservoir properties, and construction of three-dimensional geo-models.

Training feedforward networks using simultaneous perturbation with dynamic tunneling

An efficient technique, namely simultaneous perturbation with dynamic tunneling for training single hidden layer feedforward network, is proposed. A sigmoidal hidden neuron is added to the single hidden layer neural network after training. Then the cascaded network is trained again using simultaneous perturbation. The dynamic tunneling technique is employed to detrap the local minima in training. The proposed technique is shown to give better convergence results for the selected problems, namely neuro-controller, encoder, adder, demultiplexer, XOR and L–T character recognition problem.

Application of Sequential Learning Neural Networks to Civil Engineering Modeling Problems

A sequential orthogonal approach to the building and training of a single hidden layer neural network is presented in this paper. The Sequential Learning Neural Network (SLNN) model proposed by Zhang and Morris [1]is used in this paper to tackle the common problem encountered by the conventional Feed Forward Neural Network (FFNN) in determining the network structure in the number of hidden layers and the number of hidden neurons in each layer. The procedure starts with a single hidden neuron and sequentially increases in the number of hidden neurons until the model error is sufficiently small. The classical Gram–Schmidt orthogonalization method is used at each step to form a set of orthogonal bases for the space spanned by output vectors of the hidden neurons. In this approach it is possible to determine the necessary number of hidden neurons required. However, for the problems investigated in this paper, one hidden neuron itself is sufficient to achieve the desired accuracy. The neural network architecture has been trained and tested on two practical civil engineering problems – soil classification, and the prediction o strength and workability of high performance concrete.

Simultaneous perturbation for single hidden layer networks — cascade learning

A simultaneous perturbation approach for cascade learning of single hidden layer neural network is presented. A sigmoidal hidden neuron is added to the single layer of hidden neurons after training until the error has stopped decreasing after a certain limit. Then, the cascaded network is again trained using simultaneous perturbation. Perturbation employed on the weights connecting to hidden neurons are changed to detrap the local minima in training. The proposed technique gives better convergence results for the selected problems, namely neuro-controller, XOR, L–T character recognition, two spirals, simple interaction function, harmonic function and complicated interaction function.

Comments on "Classification ability of single hidden layer feedforward neural networks"

A recent paper addresses a certain classification problem, and concludes that classification can be achieved using a single hidden layer neural network. We note here that conclusions along similar lines in a more general setting were reached in an earlier paper.

Selecting Optimal Interlinings with a Neural Network

This paper reports on the construction of an integrated tool consisting of a neural network and subjoined local approximation technique for application to the sewing process, especially for selecting optimal interlinings for woolen fabrics. A single hidden layer neural network is constructed with five input nodes, ten hidden nodes, and two output nodes. To train the network with a back-propagation learning algorithm, the mechanical parameters used as inputs for the fabrics are tensile energy, bending rigidity, bending hysteresis, shear stiffness, and shear hysteresis, while mechanical parameters used as outputs for the interlinings are bending rigidity and shear stiffness, all of which are measured on the KES-FB system. Even though the back-propagation algorithm has a higher learning accuracy and can be successfully used to select the appropriate interlining, its learning process is too slow and it gets stuck in a local minimum. This research presents a few methods for improving the efficiency of the learning process. The raw data from the KES-FB system are nonlinearly normalized, and input orders are randomized. This proce dure produces a good result, such that the error values of the prediction model are low despite the relatively small data set for training. After training, the optimal interlinings for unknown fabrics can be correctly selected through mapping and the local approximation method.

Adaptive Neurocontrol of Simulated Rotor Vibrations Using Trailing Edge Flaps

Smart structure activated trailing edge flaps are capable of actively altering the aerodynamic loads on rotor blades. Coupled with a suitable feedback control law, such actuators could potentially be used to counter the vibrations induced by periodic aerodynamic loading on the blades, without the bandwidth constraints and with a potential of lower weight penalties incurred by servo actuation methods. This paper explores new, robust individual blade control (IBC) methodologies for vibration suppression using a piezoactuated trailing edge flap. The controllers employ a single hidden layer neural network, learning in real time, to adaptively cancel the effects of periodic aerodynamic loads on the blades, greatly attenuating the resulting vibrations. Both collocated and noncollocated sensor/actuator pairs are considered. Proofs of the stability and convergence of the proposed neurocontrol strategies are provided, and numerical simulation results for a one-eighth Froude scale blade model are given which demonstrate that the controller can nearly eliminate the blade vibration arising from a wide variety of unknown, periodic disturbance sources.