Hidden Layer Nodes Research Articles

An approach to estimate EEG power spectrum as an index of heat stress using backpropagation artificial neural network

A method has been presented for an effective application of backpropagation artificial neural network (ANN) in establishment of electro-encephalogram (EEG) power spectra as an index of stress in hot environment. The power spectrum data for slow wave sleep (SWS), rapid eye movement (REM) sleep and awake (AWA) states in three groups of rats (acute heat stress, chronic heat stress and the normal) were tested by an ANN, containing 60 nodes in input layer, weighted from power spectrum data from 0 to 30 Hz, 18 nodes in hidden layer and an output node. The target output values for this network were determined with another five-layered neural network (with the structure of 3-12-1-12-3). The input and output of this network was assigned with the three well-established heat stress indices (body temperature, body weight and plasma corticosterone). The most important feature for acute stress, chronic stress and normal conditions were extracted from the third layer single neuron and used for the target value for the three-layered neural network. The ANN was found effective in recognising the EEG power spectra with an average of 96.67% for acute heat stress, 97.17% for chronic heat stress and 98.5% for normal subjects.

Utilidad de las redes neuronales artificiales en la predicción de cáncer de próstata en la biopsia transrectal

To determine whether the development of an artificial neural network (ANN) made up of clinical variables allows for the prediction of prostate biopsy (PB) outcome. Patients (n=953) underwent PB at the Arquitecto Marcide Hospital in Ferrol (Spain), between january 2000 and june 2005. The variables studied were age, PSA, digital rectal examination (DRE) and prostate volume, data for all of which were available in 843 cases. In order to determine factors related to prostate cancer (PC) diagnosis, a logistic regression analysis and a feed-forward neural network were developed, including three hidden layer nodes and an output node, representing the probability of PC. Both models were constructed from a random sample of n=643 patients (derivation set). The predictive capacity was assessed with the remaining 200 patients (validation set), by means of ROC curves and the area under the curve (AUC). PC was detected in 500 (59.3%) cases. Adjusting for age, PSA, digital rectal examination and prostate volume, in a multivariate logistic regression model it was observed that all the variables were independent predictors of PC. The AUC were 0.693 for PSA, 0.707 for prostate volume, 0.815 for logistic regression and 0.819 for ANN. The predictive capacity of the ANN was significantly higher than that of the PSA (p=0.002) and prostate volume (p<0,001) and similar to that of logistic regression (p=0.760). The ANN shows a PC prediction capacity that is significantly higher than unimodal diagnosis methods, and similar to that of logistic regression.

Utilidad de las redes neuronales artificiales en la predicción de cáncer de próstata en la biopsia transrectal

ObjectiveTo determine whether the development of an artificial neural network (ANN) made up of clinical variables allows for the prediction of prostate biopsy (PB) outcome. Material And MethodsPatients (n=953) underwent PB at the Arquitecto Marcide Hospital in Ferrol (Spain), between january 2000 and june 2005. The variables studied were age, PSA, digital rectal examination (DRE) and prostate volume, data for all of which were available in 843 cases. In order to determine factors related to prostate cancer (PC) diagnosis, a logistic regression analysis and a feed–forward neural network were developed, including three hidden layer nodes and an output node, representing the probability of PC. Both models were constructed from a random sample of n=643 patients (derivation set). The predictive capacity was assessed with the remaining 200 patients (validation set), by means of ROC curves and the area under the curve (AUC). ResultsPC was detected in 500 (59.3%) cases. Adjusting for age, PSA, digital rectal examination and prostate volume, in a multivariate logistic regression model it was observed that all the variables were independent predictors of PC. The AUC were 0.693 for PSA, 0.707 for prostate volume, 0.815 for logistic regression and 0.819 for ANN. The predictive capacity of the ANN was significantly higher than that of the PSA (p=0.002) and prostate volume (p < 0,001) and similar to that of logistic regression (p=0.760). ConclusionsThe ANN shows a PC prediction capacity that is significantly higher than unimodal diagnosis methods, and similar to that of logistic regression.

Investigating the Fault Tolerance of Neural Networks

Particular levels of partial fault tolerance (PFT) in feedforward artificial neural networks of a given size can be obtained by redundancy (replicating a smaller normally trained network), by design (training specifically to increase PFT), and by a combination of the two (replicating a smaller PFT-trained network). This letter investigates the method of achieving the highest PFT per network size (total number of units and connections) for classification problems. It concludes that for non-toy problems, there exists a normally trained network of optimal size that produces the smallest fully fault-tolerant network when replicated. In addition, it shows that for particular network sizes, the best level of PFT is achieved by training a network of that size for fault tolerance. The results and discussion demonstrate how the outcome depends on the levels of saturation of the network nodes when classifying data points. With simple training tasks, where the complexity of the problem and the size of the network are well within the ability of the training method, the hidden-layer nodes operate close to their saturation points, and classification is clean. Under such circumstances, replicating the smallest normally trained correct network yields the highest PFT for any given network size. For hard training tasks (difficult classification problems or network sizes close to the minimum), normal training obtains networks that do not operate close to their saturation points, and outputs are not as close to their targets. In this case, training a larger network for fault tolerance yields better PFT than replicating a smaller, normally trained network. However, since fault-tolerant training on its own produces networks that operate closer to their linear areas than normal training, replicating normally trained networks ultimately leads to better PFT than replicating fault-tolerant networks of the same initial size.

A NOVEL PID-LIKE NEURAL NETWORK CONTROLLER

In this paper, a novel PID-like neural network controller (PIDNNC) is created. It is composed of a neural network with no more than 3 neural nodes in hidden layer, and there are an activation feedback and (or) an output feedback in hidden layer, respectively. This special structure makes the network be able to be a P, PI, PD, or PID controller as needed. The proposed controller weights can be updated on-line according to errors caused by non-linear and uncertain factors of system, based on some adaptation mechanism. The resilient back-propagation algorithm with sign instead of the gradient is used to derive the rule of updating network weights. The basic ideas, techniques and analysis are presented. Finally, we give the simulation experiment of a double inverted-pendulum system and the comparison of the effects between the proposed control strategy and the conventional one.

Simulating size effect on shear strength of RC beams without stirrups using neural networks

An artificial neural network (ANN) model was developed using past experimental data on shear failure of slender RC beams without web reinforcements. The neural network model has five input nodes representing the concrete compressive strength ( f′ c), beam width ( b), effective depth ( d), shear span to depth ratio ( a/ d), longitudinal steel ratio ( ρ), five hidden layer nodes and one output node representing the ultimate shear strength ( v u= V u/ bd). The model gives reasonable predictions of the ultimate shear stress and can simulate the size effect on ultimate shear stress at diagonal tension failure. The ANN model performs well when compared with existing empirical, theoretical and design code equations. Through the parametric studies using the ANN model, the effects of various parameters such as f′ c, d, ρ and a/ d on the shear capacity of RC beams without web reinforcement was shown. This shows the versatility of ANNs in constructing relationships among multiple variables of complex physical processes using actual experimental data for training.

Use of genetic algorithms for neural networks to predict community-acquired pneumonia

Background: Genetic algorithms have been used to solve optimization problems for artificial neural networks (ANN) in several domains. We used genetic algorithms to search for optimal hidden-layer architectures, connectivity, and training parameters for ANN for predicting community-acquired pneumonia among patients with respiratory complaints. Methods: Feed-forward back-propagation ANN were trained on sociodemographic, symptom, sign, comorbidity, and radiographic outcome data among 1044 patients from the University of Illinois (the training cohort), and were applied to 116 patients from the University of Nebraska (the testing cohort). Binary chromosomes with genes representing network attributes, including the number of nodes in the hidden layers, learning rate and momentum parameters, and the presence or absence of implicit within-layer connectivity using a competition algorithm, were operated on by various combinations of crossover, mutation, and probabilistic selection based on network mean-square error (MSE), and separately on average cross entropy (ENT). Predictive accuracy was measured as the area under a receiver-operating characteristic (ROC) curve. Results: Over 50 generations, the baseline genetic algorithm evolved an optimized ANN with nine nodes in the first hidden layer, zero nodes in the second hidden layer, learning rate and momentum parameters of 0.5, and no within-layer competition connectivity. This ANN had an ROC area in the training cohort of 0.872 and in the testing cohort of 0.934 ( P-value for difference, 0.181). Algorithms based on cross-generational selection, Gray coding of genes prior to mutation, and crossover recombination at different genetic levels, evolved optimized ANN identical to the baseline genetic strategy. Algorithms based on other strategies, including elite selection within generations (training ROC area 0.819), and inversions of genetic material during recombination (training ROC area 0.812), evolved less accurate ANN. Conclusion: ANN optimized by genetic algorithms accurately discriminated pneumonia within a training cohort, and within a testing cohort consisting of cases on which the networks had not been trained. Genetic algorithms can be used to implement efficient search strategies for optimal ANN to predict pneumonia.

Application of neural networks in forecasting engine systems reliability

This paper presents a comparative study of the predictive performances of neural network time series models for forecasting failures and reliability in engine systems. Traditionally, failure data analysis requires specifications of parametric failure distributions and justifications of certain assumptions, which are at times difficult to validate. On the other hand, the time series modeling technique using neural networks provides a promising alternative. Neural network modeling via feed-forward multilayer perceptron (MLP) suffers from local minima problems and long computation time. The radial basis function (RBF) neural network architecture is found to be a viable alternative due to its shorter training time. Illustrative examples using reliability testing and field data showed that the proposed model results in comparable or better predictive performance than traditional MLP model and the linear benchmark based on Box–Jenkins autoregressive-integrated-moving average (ARIMA) models. The effects of input window size and hidden layer nodes are further investigated. Appropriate design topologies can be determined via sensitivity analysis.

Designing Compact Feedforward Neural Models with Small Training Data Sets

A new hybrid method is presented for designing feedforward, backpropagation neural models with small training data sets. The method minimizes the generalization error, a fundamental quantity that characterizes the effectiveness of the regression models. It combines into one framework a bootstrap technique that estimates network generalization performance and a collection of stochastic and deterministic optimization techniques that adjust neural network interconnection geometry. The approach is derived as a form of multi-objective optimization strategy. This allows for more direct treatment of contradictory design criteria than traditionally employed single-objectivetechniques.Astochasticoptimization methodsuch asageneticalgorithmisusedto selectactivation functions for hidden-layer nodes, whereas fast deterministic techniques, optimal brain surgeon and singular value decomposition, are used to perform connection and node pruning. The method is demonstrated by optimizing neural networks that model the high-lift aerodynamics of a multi-element airfoil. The neural model is constructed using a small computational data set consisting of 227 data points. In the numerical experiments presented, the solutions produced by this hybrid approach exhibit an improvement in the generalization ability on the average of e ve to six times when compared to the pruned models with only one type of activation function. When traditional fully connected networks with hyperbolic tangent activation functions are considered, the improvement in the generalization performance of the new models is even greater. The neural models exhibit superior generalization qualities that are virtually impossible to e nd by manual trial-and-error approaches.

Comparison of Four Artificial Neural Network Software Programs Used to Predict the In Vitro Dissolution of Controlled-Release Tablets

The purpose of this study was to evaluate four commercially available artificial neural network (ANN) software programs: NeuroShell2 v3.0, BrainMaker v3.7, CAD/Chem v5.0, and NeuralWorks Professional II/Plus for prediction of in vitro dissolution-time profiles of controlled-release tablets containing a model sympathomimetic drug. Seven independent formulation variables and three other tablet variables (moisture content of granules, granule particle size, and tablet hardness), for 22 tablet formulations, were used as the ANN model input. In vitro dissolution time-profiles at 10 different sampling times were used as the output. The models' optimum architectures were determined for each ANN software by varying the number of hidden layers and number of nodes in hidden layer(s). The ANN developed from the four software programs were validated by predicting the in vitro dissolution time-profiles of each of the 19 formulations, which were excluded from the training process. Although the same data set was used, the optimum ANN architectures generated from the four software programs were different. Using the four optimum ANN models, the plots of predicted vs. observed percentage of drug dissolved gave slopes ranging from 0.95 to 1.01 and r2 values ranging from 0.95 to 0.99 for all 190 dissolution data points for the 19 training formulations. The difference factors (f1) and similarity factors (f2) between the ANN predicted and the observed in vitro dissolution profiles were also used to compare the predictions for the four software programs. It was concluded that the four programs provided reasonable predictions of in vitro dissolution profiles for the data set employed in this study, with NeuralShell2 showing the best overall prediction.

Of overparametrization in non-linear system identification and neural networks

The effects of overparametrization in non-linear system identification and neural networks are often presented as overfitting to a particular data set. An overparametrized model tends to interpolate the data set and can fail to capture the underlying dynamics represented by the data. Past studies have usually concentrated on overparametrisation in the number of model terms or the number of hidden layer nodes. In the present study, the effects of several forms of overparametrization on the model dynamics are compared using qualitative validation methods. The dynamic properties of the identified model or the trained network are graphically displayed and compared with those of the real system. It is shown that overparametrization in the number of time lags or input nodes has the worst effect on the dynamic properties of the resulting model.

A dynamical model for the analysis and acceleration of learning in feedforward networks

A dynamical system model is derived for feedforward neural networks with one layer of hidden nodes. The model is valid in the vicinity of flat minima of the cost function that rise due to the formation of clusters of redundant hidden nodes with nearly identical outputs. The derivation is carried out for networks with an arbitrary number of hidden and output nodes and is, therefore, a generalization of previous work valid for networks with only two hidden nodes and one output node. The Jacobian matrix of the system is obtained, whose eigenvalues characterize the evolution of learning. Flat minima correspond to critical points of the phase plane trajectories and the bifurcation of the eigenvalues signifies their abandonment. Following the derivation of the dynamical model, we show that identification of the hidden nodes clusters using unsupervised learning techniques enables the application of a constrained application (Dynamically Constrained Back Propagation—DCBP) whose purpose is to facilitate prompt bifurcation of the eigenvalues of the Jacobian matrix and, thus, accelerate learning. DCBP is applied to standard benchmark tasks either autonomously or as an aid to other standard learning algorithms in the vicinity of flat minima. Its application leads to significant reduction in the number of required epochs for convergence.

Learning efficiency improvement of back-propagation algorithm by error saturation prevention method

Back-propagation (BP) algorithm is currently the most widely used learning algorithm in artificial neural networks. With proper selection of feed-forward neural network architecture, it is capable of approximating most problems with high accuracy and generalization ability. However, the slow convergence is a serious problem when using this well-known BP learning algorithm in many applications. As a result, many researchers take effort to improve the learning efficiency of BP algorithm by various enhancements. In this research, we consider that the error saturation (ES) condition, which is caused by the use of gradient descent method, will greatly slow down the learning speed of BP algorithm. Thus, in this paper, we will analyze the causes of the ES condition in the output layer. An error saturation prevention (ESP) function is then proposed to prevent the nodes in the output layer from the ES condition. We also apply this method to the nodes in hidden layers to adjust the learning terms. By the proposed methods, we can not only improve the learning efficiency by the ES condition prevention but also maintain the semantic meaning of the energy function. Finally, some simulations are given to show the workings of our proposed method.

Construction of a new adaptive wavelet network and its learning algorithm

A new adaptive learning algorithm for constructing and training wavelet networks is proposed based on the time-frequency localization properties of wavelet frames and the adaptive projection algorithm. The exponential convergence of the adaptive projection algorithm in finite-dimensional Hilbert spaces is constructively proved, with exponential decay ratios given with high accuracy. The learning algorithm can sufficiently utilize the time-frequency information contained in the training data, iteratively determines the number of the hidden layer nodes and the weights of wavelet networks, and solves the problem of structure optimization of wavelet networks. The algorithm is simple and efficient, as illustrated by examples of signal representation and denoising.

Mobile radio propagation prediction for a urban microcellular environment using factorizable Gaussian neural networks

In this paper, we present the application of a factorizable Gaussian neural network to the prediction of field strength in an urban microcellular environment. The Gaussian neural network is a two-layer localized receptive field network whose output nodes form a combination of Gaussian radial activation functions computed by the hidden layer nodes. Appropriate centres, spreads and connection weights in the Gaussian network lead to a network that is capable of forming the best approximation to any continuous nonlinear mapping up to an arbitrary resolution. Such an approximation introduces the best nonlinear approximation capability into the prediction model in order to predict propagation loss accurately over an arbitrary environment based on adaptive learning from measurement data. The adaptive learning employs a gradientdescent algorithm with a combination of both the delta-bar-delta rule and momentum heuristics to enhance its convergence performance. The applications to Lee's field measurements taken in Irvine, CA, USA, are conducted to demonstrate the effectiveness of the Gaussian neural network approach.

A CONSTRUCTIVE NEURAL NETWORK ALGORITHM FOR FUNCTION APPROXIMATION USING LOCALLY FIT SIGMOIDS

A study of the function approximation capabilities of single hidden layer neural networks strongly motivates the investigation of constructive learning techniques as a means of realizing established error bounds. Learning characteristics employed by constructive algorithms provide ideas for development of new algorithms applicable to the function approximation problem. In addition, constructive techniques offer efficient methods for network construction and weight determination. The development of a novel neural network algorithm, the Constructive Locally Fit Sigmoids (CLFS) function approximation algorithm, is presented in detail. Basis functions of global extent (piecewise linear sigmoidal functions) are locally fit to the target function, resulting in a pool of candidate hidden layer nodes from which a function approximation is obtained. This algorithm provides a methodology of selecting nodes in a meaningful way from the infinite set of possibilities and synthesizes an n node single hidden layer network with empirical and analytical results that strongly indicate an O(1/n) mean squared training error bound under certain assumptions. The algorithm operates in polynomial time in the number of network nodes and the input dimension. Empirical results demonstrate its effectiveness on several multidimensional function approximate problems relative to contemporary constructive and nonconstructive algorithms.

Using Quasirandom Weights in Neural Networks

ABSTRACTWe present a novel training algorithm for a feed forward neural network with a single hidden layer of nodes (i.e., two layers of connection weights). Our algorithm is capable of training networks for hard problems, such as the classic two-spirals problem. The weights in the first layer are determined using a quasirandom number generator. These weights are frozen—they are never modified during the training process. The second layer of weights is trained as a simple linear discriminator using methods such as the pseudoinverse, with possible iterations. We also study the problem of reducing the hidden layer: pruning low-weight nodes and a genetic algorithm search for good subsets.

Ontogenic neuro-fuzzy algorithm: F-CID3

The paper introduces ontogenic Fuzzy-CID3 algorithm (F-CID3) which combines a neural network algorithm and fuzzy sets into a single hybrid algorithm which generates its own topology. Two new methods, one based on a concept of a neural fuzzy number tree, and a class separation method are introduced in the paper and utilized in the algorithm. The F-CID3 algorithm is an extension of an ontogenic CID3 algorithm which generates a neural network architecture by minimizing Shannon's entropy function. The F-CID3 algorithm generates an initial network architecture in the same way as the CID3 algorithm. It subsequently defines grades of membership for fuzzy sets associated with hidden layer nodes where the entropy is first reduced to zero, and then switches entirely to operations on fuzzy sets. This hybrid approach results in a simpler architecture realization than the CID3, with fewer connections. The performance of the algorithm is analyzed on benchmark examples.

A Neural Network Approach to Tracking Eye Position

The design of a neural network based eye tracker is presented. A series of experiments with counterpropagation neural networks convert synthetic video images into eye coordinates by an enhanced feed-forward neural network with multiple winning hidden layer nodes. Difficulties encountered during the design process are discussed. The results show that accurate, fine-grained tracking of a human's eye position is possible by processing the video image collected from a goggle-mounted miniature charge-coupled device (CCD) camera.

Environment-adaptation mobile radio propagation prediction using radial basis function neural networks

This paper investigates the application of a radial basis function (RBF) neural network to the prediction of field strength based on topographical and morphographical data. The RBF neural network is a two-layer localized receptive field network whose output nodes from a combination of radial activation functions computed by the hidden layer nodes. Appropriate centers and connection weights in the RBF network lead to a network that is capable of forming the best approximation to any continuous nonlinear mapping up to an arbitrary resolution. Such an approximation introduces best nonlinear approximation capability into the prediction model in order to accurately predict propagation loss over an arbitrary environment based on adaptive learning from measurement data. The adaptive learning employs hybrid competitive and recursive least squares algorithms. The unsupervised competitive algorithm adjusts the centers while the recursive least squares (RLS) algorithm estimates the connection weights. Because these two learning rules are both linear, rapid convergence is guaranteed. This hybrid algorithm significantly enhances the real-time or adaptive capability of the RBF-based prediction model. The applications to Okumura's (1968) data are included to demonstrate the effectiveness of the RBF neural network approach.