Hidden Units Research Articles

Neural network tomography: Network replication from output surface geometry

Multilayer perceptron networks whose outputs consist of affine combinations of hidden units using the tanh activation function are universal function approximators and are used for regression, typically by reducing the MSE with backpropagation. We present a neural network weight learning algorithm that directly positions the hidden units within input space by numerically analyzing the curvature of the output surface. Our results show that under some sampling requirements, this method can reliably recover the parameters of a neural network used to generate a data set.

An Iterative Method for Deciding SVM and Single Layer Neural Network Structures

We present two new classifiers for two-class classification problems using a new Beta-SVM kernel transformation and an iterative algorithm to concurrently select the support vectors for a support vector machine (SVM) and the hidden units for a single hidden layer neural network to achieve a better generalization performance. To construct the classifiers, the contributing data points are chosen on the basis of a thresholding scheme of the outputs of a single perceptron trained using all training data samples. The chosen support vectors are used to construct a new SVM classifier that we call Beta-SVN. The number of chosen support vectors is used to determine the structure of the hidden layer in a single hidden layer neural network that we call Beta-NN. The Beta-SVN and Beta-NN structures produced by our method outperformed other commonly used classifiers when tested on a 2-dimensional non-linearly separable data set.

Research on artificial neural networks with spatial architecture based on span connection and lateral inhibition mechanism

Motivated by the knowledge of biological neural system being an asymmetry three dimensional structure and the effect of inhibition within cerebral cortex, we propose a novel topology of artificial neural network called spatial artificial neural network (SANN), which includes two types of processing networks: basic network and spatial connection network. The basic network introduces the lateral inhibition mechanism between hidden units and realises the competition in neurons. The spatial connection means that any two neurons in SANN may have random and long-range connectivity. Supervised learning rules for synaptic weights update are derived from the steepest descent gradient, and the descent gradient with momentum (GDM) is used for network learning. From the experimental analysis of benchmark problems such as pattern recognition, non-linear function approximation, we prove the powerful representation capability and generalisation performance of SANN network.

Learning algorithm in restricted Boltzmann machines using Kullback-Leibler importance estimation procedure

Restricted Boltzmann machines (RBMs) are bipartite structured statistical neural networks and consist of two layers. One of them is a layer of visible units and the other one is a layer of hidden units. In each layer, any units do not connect to each other. RBMs have high flexibility and rich structure and have been expected to applied to various applications, for example, image and pattern recognitions, face detections and so on. However, most of computational models in RBMs are intractable and often belong to the class of NP-hard problem. In this paper, in order to construct a practical learning algorithm for them, we employ the Kullback-Leibler Importance Estimation Procedure (KLIEP) to RBMs, and give a new scheme of practical approximate learning algorithm for RBMs based on the KLIEP.

Prediction for Relative Dynamic Elastic Modulus of PVA-ECC under Freezing and Thawing Cycles

This study focuses on the relative dynamic elastic modulus of the polyvinyl alcohol fiber reinforced cementitious composite (PVA-ECC) after three hundred freeze-thaw cycles. The Artificial Neural Network of freeze-thaw cycles prediction was finally established through data analysis with the help of BP artificial neural network, calculation method optimization and sample training, many times’ trails of the hidden layer and every hidden unit, and the optimal selection of the training function. The results show that there is a small relative error between the predicted value and the actual one of the specimen of the relative dynamic elastic modulus of the PVA-ECC, and the established artificial neural network model bears a higher prediction precision.

AN ONLINE SELF-ORGANIZING SCHEME FOR PARSIMONIOUS AND ACCURATE FUZZY NEURAL NETWORKS

In this paper, an online self-organizing scheme for Parsimonious and Accurate Fuzzy Neural Networks (PAFNN), and a novel structure learning algorithm incorporating a pruning strategy into novel growth criteria are presented. The proposed growing procedure without pruning not only simplifies the online learning process but also facilitates the formation of a more parsimonious fuzzy neural network. By virtue of optimal parameter identification, high performance and accuracy can be obtained. The learning phase of the PAFNN involves two stages, namely structure learning and parameter learning. In structure learning, the PAFNN starts with no hidden neurons and parsimoniously generates new hidden units according to the proposed growth criteria as learning proceeds. In parameter learning, parameters in premises and consequents of fuzzy rules, regardless of whether they are newly created or already in existence, are updated by the extended Kalman filter (EKF) method and the linear least squares (LLS) algorithm, respectively. This parameter adjustment paradigm enables optimization of parameters in each learning epoch so that high performance can be achieved. The effectiveness and superiority of the PAFNN paradigm are demonstrated by comparing the proposed method with state-of-the-art methods. Simulation results on various benchmark problems in the areas of function approximation, nonlinear dynamic system identification and chaotic time-series prediction demonstrate that the proposed PAFNN algorithm can achieve more parsimonious network structure, higher approximation accuracy and better generalization simultaneously.

Repeatability in computer‐aided diagnosis: Application to breast cancer diagnosis on sonography

The aim of this study was to investigate the concept of repeatability in a case-based performance evaluation of two classifiers commonly used in computer-aided diagnosis in the task of distinguishing benign from malignant lesions. The authors performed .632+ bootstrap analyses using a data set of 1251 sonographic lesions of which 212 were malignant. Several analyses were performed investigating the impact of sample size and number of bootstrap iterations. The classifiers investigated were a Bayesian neural net (BNN) with five hidden units and linear discriminant analysis (LDA). Both used the same four input lesion features. While the authors did evaluate classifier performance using receiver operating characteristic (ROC) analysis, the main focus was to investigate case-based performance based on the classifier output for individual cases, i.e., the classifier outputs for each test case measured over the bootstrap iterations. In this case-based analysis, the authors examined the classifier output variability and linked it to the concept of repeatability. Repeatability was assessed on the level of individual cases, overall for all cases in the data set, and regarding its dependence on the case-based classifier output. The impact of repeatability was studied when aiming to operate at a constant sensitivity or specificity and when aiming to operate at a constant threshold value for the classifier output. The BNN slightly outperformed the LDA with an area under the ROC curve of 0.88 versus 0.85 (p < 0.05). In the repeatability analysis on an individual case basis, it was evident that different cases posed different degrees of difficulty to each classifier as measured by the by-case output variability. When considering the entire data set, however, the overall repeatability of the BNN classifier was lower than for the LDA classifier, i.e., the by-case variability for the BNN was higher. The dependence of the by-case variability on the average by-case classifier output was markedly different for the classifiers. The BNN achieved the lowest variability (best repeatability) when operating at high sensitivity (> 90%) and low specificity (< 66%), while the LDA achieved this at moderate sensitivity (approximately 74%) and specificity (approximately 84%). When operating at constant 90% sensitivity or constant 90% specificity, the width of the 95% confidence intervals for the corresponding classifier output was considerable for both classifiers and increased for smaller sample sizes. When operating at a constant threshold value for the classifier output, the width of the 95% confidence intervals for the corresponding sensitivity and specificity ranged from 9 percentage points (pp) to 30 pp. The repeatability of the classifier output can have a substantial effect on the obtained sensitivity and specificity. Knowledge of classifier repeatability, in addition to overall performance level, is important for successful translation and implementation of computer-aided diagnosis in clinical decision making.

Temporal-Kernel Recurrent Neural Networks

A Recurrent Neural Network (RNN) is a powerful connectionist model that can be applied to many challenging sequential problems, including problems that naturally arise in language and speech. However, RNNs are extremely hard to train on problems that have long-term dependencies, where it is necessary to remember events for many timesteps before using them to make a prediction. In this paper we consider the problem of training RNNs to predict sequences that exhibit significant long-term dependencies, focusing on a serial recall task where the RNN needs to remember a sequence of characters for a large number of steps before reconstructing it. We introduce the Temporal-Kernel Recurrent Neural Network (TKRNN), which is a variant of the RNN that can cope with long-term dependencies much more easily than a standard RNN, and show that the TKRNN develops short-term memory that successfully solves the serial recall task by representing the input string with a stable state of its hidden units.

An Integral Upper Bound for Neural Network Approximation

Complexity of one-hidden-layer networks is studied using tools from nonlinear approximation and integration theory. For functions with suitable integral representations in the form of networks with infinitely many hidden units, upper bounds are derived on the speed of decrease of approximation error as the number of network units increases. These bounds are obtained for various norms using the framework of Bochner integration. Results are applied to perceptron networks.

Modelagem do sistema de resfriamento por imersão de carcaças de frangos utilizando redes neurais artificiais

The process of cooling chicken carcasses by immersing them in mixture of cold water and ice (chillers) is complex. It is very difficult to represent it by a transport phenomenon model. In this work, artificial neural networks were used with an intermediary layer in the description and modeling of the cooling process of chickens. Different architectures of the neural network were tested, altering the numbers of input and hidden units. Data supplied by the Sadia-Toledo Company were used for training and validation of the neural networks. The input variables selected for the model were the following: carcass weight, initial temperature, propylene glycol temperature with external circulation, water flow rate of water in each tank, renewal water cooling time and temperature, and as output variable the temperature of the chicken when exiting the chiller. The results obtained showed that the network with 8 neurons in the input layer and 24 in the hidden layer best represented the investigated system.

Estimating printed circuit board assembly times using neural networks

Several production planning tasks in the printed circuit board (PCB) assembly industry involve the estimation of the component placement times for different PCB types and placement machines. This kind of task may be, for example, the scheduling of jobs or line balancing for single or multiple jobs. The simplest approach to time estimation is to let the production time be a linear function of the number of components to be placed. To achieve more accurate results, the model should include more parameters (e.g. the number of different component types, the number of different component shapes, the dimensions of the PCBs, etc.). In this study we train multilayer neural networks to approximate the assembly times of two different types of assembly machines based on several parameter combinations. It turns out that conventional learning methods are prone to overfitting when the number of hidden units of the network is large in relation to the number of training cases. To avoid this and complicated training and testing, we use Bayesian regularisation to achieve efficient learning and good accuracy automatically.

Traffic congestion identification by combining PCA with higher-order Boltzmann machine

Traffic congestion identification is a popular research topic of intelligent transport systems. Identification rate of existing methods usually cannot meet the practical requirements. To improve the identification rate and reduce the computation cost, a novel intelligent identification method is proposed. The proposed method combines principal component analysis (PCA) method with higher-order Boltzmann machine (BM). PCA is used to reduce the dimension of input feature space. It can not only reduce the computation cost but also filter noise of the source data. BM is a kind of stochastic network that can obtain the global optimum solution. Higher-order BM without hidden units can save lots of computation cost without decreasing modeling power. The trained higher-order BM is used to identify traffic state. The efficiency of the proposed method is illustrated through analyzing Jinan urban transportation data.

AN INCREMENTAL FRAMEWORK BASED ON CROSS-VALIDATION FOR ESTIMATING THE ARCHITECTURE OF A MULTILAYER PERCEPTRON

We define the problem of optimizing the architecture of a multilayer perceptron (MLP) as a state space search and propose the MOST (Multiple Operators using Statistical Tests) framework that incrementally modifies the structure and checks for improvement using cross-validation. We consider five variants that implement forward/backward search, using single/multiple operators and searching depth-first/breadth-first. On 44 classification and 30 regression datasets, we exhaustively search for the optimal and evaluate the goodness based on: (1) Order, the accuracy with respect to the optimal and (2) Rank, the computational complexity. We check for the effect of two resampling methods (5 × 2, ten-fold cv), four statistical tests (5 × 2 cv t, ten-fold cv t, Wilcoxon, sign) and two corrections for multiple comparisons (Bonferroni, Holm). We also compare with Dynamic Node Creation (DNC) and Cascade Correlation (CC). Our results show that: (1) On most datasets, networks with few hidden units are optimal, (2) forward searching finds simpler architectures, (3) variants using single node additions (deletions) generally stop early and get stuck in simple (complex) networks, (4) choosing the best of multiple operators finds networks closer to the optimal, (5) MOST variants generally find simpler networks having lower or comparable error rates than DNC and CC.

A flexible visual inspection system based on neural networks

In this work, we propose a neural networks-based machine vision system, which is intended to act as a reconfigurable inspection tool, for use in manufacturing environments. The processing engine of the system is a second-order neural network, which extracts geometric features invariant to translation and rotation. A major issue with the use of higher-order neural networks is the combinatorial explosion of the higher-order terms, which is addressed here with the use of the alternative image representation strategy of coarse coding. We developed a genetic algorithms tool, which allows the automated determination of the optimal number of hidden units in the neural networks architecture. The inspection system is tested in two application areas, namely inspection of axisymmetric components and classification of rivets. Numerous tests are carried out to evaluate the robustness of the proposed system to complex noise sources.

A Local Linear Wavelet Neural Network Based on a Bayesian Design Method

In general, wavelet neural networks have a problem on the curse of dimensionality, i.e. hidden units to be required are exponentially increased with a high input dimension. To solve the above problem, the wavelet neural network incorporating local linear model is proposed. On the network design, however, the number of hidden units are determined by trial and error. In the present paper, a design method based on Bayesian method is proposed for the local linear wavelet neural network. Through computer simulation, the performance of the proposed method is evaluated.

Artificial Neural Networks that Classify Musical Chords

An artificial neural network was trained to classify musical chords into four categories—major, dominant seventh, minor, or diminished seventh—independent of musical key. After training, the internal structure of the network was analyzed in order to determine the representations that the network was using to classify chords. It was found that the first layer of connection weights in the network converted the local representations of input notes into distributed representations that could be described in musical terms as circles of major thirds and on circles of major seconds. Hidden units then were able to use this representation to organize stimuli geometrically into a simple space that was easily partitioned by output units to classify the stimuli. This illustrates one potential contribution of artificial neural networks to cognitive informatics: the discovery of novel forms of representation in systems that can accomplish intelligent tasks.

Upper bound on pattern storage in feedforward networks

An upper bound on pattern storage is stated for nonlinear feedforward networks with analytic activation functions, like the multilayer perceptron and radial basis function network. The bound is given in terms of the number of network weights, and applies to networks having any number of output nodes and arbitrary connectivity. Starting from the strict interpolation equations and exact finite degree polynomial models for the hidden units, a straightforward proof by contradiction is developed for the upper bound. Several networks, trained by conjugate gradient, are used to demonstrate the tightness of the bound for random patterns.

Boltzmann Machines with Identified States

Learning for boltzmann machines deals with each state individually. If given data is categorized, the probabilities have to be distributed to each state, not to each catetory. We propose boltzmann machines identifying the states in the same categories. Boltzmann machines with hidden units are the special cases. Boltzmann learning and em algorithm are effective learning methods for boltzmann machines. We solve boltzmann learning and em algorithm for the proposed models.

Neural Network Models for Conditional Distribution Under Bayesian Analysis

We use neural networks (NN) as a tool for a nonlinear autoregression to predict the second moment of the conditional density of return series. The NN models are compared to the popular econometric GARCH(1,1) model. We estimate the models in a Bayesian framework using Markov chain Monte Carlo posterior simulations. The interlinked aspects of the proposed Bayesian methodology are identification of NN hidden units and treatment of NN complexity based on model evidence. The empirical study includes the application of the designed strategy to market data, where we found a strong support for a nonlinear multilayer perceptron model with two hidden units.

Effects of Initialization on Rule Extraction in Structural Learning

This paper studies how our previously proposed initialization effects the rule extraction of neural networks by structural learning with forgetting. The proposed initialization consists of two steps: (1) initializing weights of hidden units so that their separation hyperplanes should pass through the center of an input pattern set and (2) initializing those of output units to zero. From simulation results on Boolean function discovery problems with 5 and 7 inputs, it has been confirmed that the proposed initialization yields a simpler network structure and higher rule extraction ability than the conventional initialization giving uniform random number to all the initial weights of the network.