Hidden Units Research Articles

PENERAPAN MODEL HYBRID ARIMA BACKPROPAGATION UNTUK PERAMALAN HARGA GABAH INDONESIA

Hybrid model discussed in this paper combining ARIMA and backpropagation is applied to grain price forecasting in Indonesia for period January 2008 until April 2013. The grain price time series consists of linear and nonlinear patterns. Backpropagations can recognize non linear patterns that can not be done by ARIMA. In order to find the best model, some combinations of prepocessing transformations, the number of input and hidden units, and the activation function were applied in the contruction of the network structure. Based on the experiments, it can be showed that ARIMA backpropagation hybrid model provides more accurate results than ARIMA model. The hybrid model would rather be used in the short-term forecasting, no more than three periods. Keywords : ARIMA, Backpropagation, Hybrid, Grain Price

Enhancing performance of the backpropagation algorithm via sparse response regularization

The backpropagation (BP) algorithm is the most commonly utilized training strategy for a feed-forward artificial neural network (FFANN). The BP algorithm, however, always leads to the problems of low convergence rate, high energy and poor generalization capability of FFANN. In this paper, motivated by the sparsity property of human neuron’ responses, we introduce a new sparse-response BP (SRBP) to improve the capacity of a FFANN by enforcing sparsity to its hidden units through imposing a supplemental L1 term on them. The FFANN model learned from our algorithm is closely related to the real human and thus its mechanism fully complies with the human nervous system, i.e., sparse representation and architectural depth. Experiments on several datasets demonstrate that SRBP yields good performances on convergence rate, energy saving and generalization capability.

Using Neural Networks to Modeling Vehicle Dynamics

This paper is the result of a research program which focused on the statistical dynamics of vehicles. Most of the inputs of man-machine-field system have a random variation, so a systemic and statistical analysis of vehicle dynamics is obvious. In our study, data were obtained by measuring the dynamic parameters of vehicles and engines. Testing program aimed to capture a large range of operating regimes. To analyze the data the authors have used neural networks. There was adopted a NNARX (Neural Network Auto-Regressive with eXogene inputs) model with 4 inputs, 5 hidden units and 1 output. It can be concluded that the development of mathematical modeling using non-linear neural network can ensure the desired accuracy, conveniently is obtained by increasing the number of neurons in the hidden laws.

Channel mapping using bidirectional long short-term memory for dereverberation in hands-free voice controlled devices

In this article, the reverberation problem for hands-free voice controlled devices is addressed by employing Bidirectional Long Short-Term Memory (BLSTM) recurrent neural networks. Such networks use memory blocks in the hidden units, enabling them to exploit a self-learnt amount of temporal context. The main objective of this technique is to minimize the mismatch between the distant talk (reverberant/distorted) speech and the close talk (clean) speech. To achieve this, the network is trained by mapping the cepstral feature space from the distant talk channel to its counterpart from the close talk channel frame-wisely in terms of regression. The method has been successfully evaluated on a realistically recorded reverberant French corpus by a large scale of experiments of comparing a variety of network architectures, investigating different network training targets (differential or absolute), and combining with common adaptation techniques. In addition, the robustness of this technique is also accessed by cross-room evaluation on both, a simulated French corpus and a realistic English corpus. Experimental results show that the proposed novel BLSTM dereverberation models trained by the differential targets reduce the word error rate (WER) by 16% relatively on the French corpus (intra room scenario) as well as 8% relatively on the English corpus (inter room scenario).

A Novel Optimized Adaptive Learning Approach of RBF on Biomedical Data Sets

In this study, we propose a novel learning approach of Radial Basis Function Neural Network (RBFNN) based on Fuzzy C-Means (FCM) and Quantum Particle Swarm Optimization (QPSO) to group similar data. The performance of RBFNN relies on the parameters such as number of hidden nodes, centres and width of Gaussian function and weight matrix between hidden layer and output layer. Generally, RBF is trained with a fixed number of nodes but in this study we allow the network to have variable number of hidden nodes based on the size of input samples. The clustering algorithm, Fuzzy C Means (FCM) is optimized with QPSO to provide global optimal centres for RBFNN. The weights are calculated by using Least Square Method and the root mean square error is optimized to improve the accuracy, accordingly the hidden unit numbers are adjusted. The cluster centres are obtained using optimized FCM and are checked against random selection of centres to verify the suitability. The datasets such as liver disorder and breast cancer from UCI machine learning repository are used for the experiments. The accuracy is analyzed for the Cluster Numbers (CN) 2, 3, 4, 5, 6, 7, 8, 9, 10, 15 and 20, respectively.

Extending the Extreme Physical Information to Universal Cognitive Models via a Confident Information First Principle

The principle of extreme physical information (EPI) can be used to derive many known laws and distributions in theoretical physics by extremizing the physical information loss K, i.e., the difference between the observed Fisher information I and the intrinsic information bound J of the physical phenomenon being measured. However, for complex cognitive systems of high dimensionality (e.g., human language processing and image recognition), the information bound J could be excessively larger than I (J ≫ I), due to insufficient observation, which would lead to serious over-fitting problems in the derivation of cognitive models. Moreover, there is a lack of an established exact invariance principle that gives rise to the bound information in universal cognitive systems. This limits the direct application of EPI. To narrow down the gap between I and J, in this paper, we propose a confident-information-first (CIF) principle to lower the information bound J by preserving confident parameters and ruling out unreliable or noisy parameters in the probability density function being measured. The confidence of each parameter can be assessed by its contribution to the expected Fisher information distance between the physical phenomenon and its observations. In addition, given a specific parametric representation, this contribution can often be directly assessed by the Fisher information, which establishes a connection with the inverse variance of any unbiased estimate for the parameter via the Cramér–Rao bound. We then consider the dimensionality reduction in the parameter spaces of binary multivariate distributions. We show that the single-layer Boltzmann machine without hidden units (SBM) can be derived using the CIF principle. An illustrative experiment is conducted to show how the CIF principle improves the density estimation performance.

Symbolic neural networks for cognitive capacities

Abstract Pattern recognition (recognizing a pattern from inputs) and recall (describing or predicting the inputs associated with a recognizable pattern) are essential for neural-symbolic processing and cognitive capacities. Without them the brain cannot interact with the world e.g.: form internal representations and recall memory upon which it can perform logic and reason. Neural networks are efficient, biologically plausible algorithms that can perform large scale recognition. However, most neural network models of recognition perform recognition but not recall: they are sub-symbolic. It remains difficult to connect models of recognition with models of logic and emulate fundamental brain functions, because of the symbolic recall limitation. I introduce a completely symbolic neural network that is similar in function to standard feedforward neural networks but uses feedforward-feedback connections similar to auto-associative networks. However, unlike auto-associative networks, the symmetrical feedback connections are inhibitory not excitatory. Initially it may seem counterintuitive that recognition can even occur because the top-down connections are self-inhibitory. The self-inhibitory configuration is used to implement a gradient descent mechanism that functions during recognition not learning. The purpose of the gradient-descent is not to learn weights (weights are still learned during learning) but to find neuron activation. The advantage of this approach is the weights can now be symbolic (representing prototypes of expected patterns) allowing recall within neural networks. Moreover, considering the costs of both learning and recognition, this approach may be more efficient than feedforward recognition. I show that this model mathematically emulates standard feedforward model equations in single layer networks without hidden units. Comparisons that include more layers are planned in the future.

Multilayer perceptrons, Hopfield’s associative memories, and restricted Boltzmann machines

This study was intended to describe multilayer perceptrons (MLP), Hopfield’s associative memories (HAM), and restricted Boltzmann machines (RBM) from a unified point of view. Despite of mutual relation between three models, for example, RBMs have been utilizing to construct deeper architectures than shallower MLPs. The energy function in HAM is analogous to the Ising model in statistical mechanics, and it connects microscopic physics to thermodynamics. The canonical partition function Z in the Boltzmann distribution is also utilized RBMs. Asynchronous updating and contrastive divergence (CD) based upon Gibbs sampling is also related. Therefore, it seems to be worth considering these three models within a common framework. This attempt might lead to “one algorithm hypothesis.”, which insists that our brains might rule a single but universal rule. An algorithm, which someone could find out in a region, may be applicable to other regions. Multilayer perceptrons (henceforth, MLP) are feed forward models for pattern recognition and classification. Hopfield proposed another kind of neural network models for associative memory and optimization (HAM). Hiton adopted the restricted Boltzmann machines (RBM) in “Deep Learning” in order to construct deeper layered neural networks. The energy employed in RBMs are elicited the generalized EM algorithm, which was closely related to the energy employed by HAM. In spite of other various differences, see Table 1, it is worth considering to compare among them. At least, an attempt is worth attempting to explain all of them in a unified terminology. HAM and RBM have symmetrically weighted connections, wij = wji, although generalized Boltzmann machines can not satisfy this constraints. Similarly, there are no feedback connections in MLP in general. When we denote a connection weight from j-th unit to i-th unit as wij, wij ∈ R, wji = 0 in MLP. When we consider a merged weight matrix W, all the models can be considered as identical. The construction methods adopted by Deep Learning are based upon RBMs. One of key concepts to success for constructing multilayer deep architecture is the non–linearity, because units in hidden layer in RBMs are binary. The non–linearity seems to play an important role to construct deep architecture. When we suppose to abandon CD and binary feature, multilayer architecture might replace one weight matrix W = W1W2... Wp. Also, we can consider a thought experiment with only one hidden unit in RBM. If h = 0, then there are no meanings at all. If h = 1, then it must be an identity mapping, or at least, it might be extract the eigenvector vector corresponded to the maximum eigenvalue value in data matrix X. This

Deep Boltzmann Machines Aided Design Based on Genetic Algorithms

To resolve the problem of no guidance about how to set the values of numerical meta-parameters and difficulty to achieve optimization of Deep Boltzmann Machines, genetic algorithms are used to develop an automatic optimizing method named GA-RBMs (Genetic Algorithm-Restricted Boltzmann Machines) for this model’s aided design. Based on the Restricted Boltzmann Machines’ features and evaluation function, a genetic algorithm is designed and realizes the global search of satisfied structure. We also initialize the network’s weights to determine the number of visible units and hidden units. The experiments were conducted on MNIST digits handwritten datasets. The results proved that this optimization reduced the dimension of visible units and improved the performance of feature extracted by Deep Boltzmann Machines. The network optimized has good generalization performance and meets the demand of Deep Boltzmann Machines’ aided design.

Memory-Enhanced Neural Networks and NMF for Robust ASR

In this article we address the problem of distant speech recognition for reverberant noisy environments. Speech enhancement methods, e. g., using non-negative matrix factorization (NMF), are succesful in improving the robustness of ASR systems. Furthermore, discriminative training and feature transformations are employed to increase the robustness of traditional systems using Gaussian mixture models (GMM). On the other hand, acoustic models based on deep neural networks (DNN) were recently shown to outperform GMMs. In this work, we combine a state-of-the art GMM system with a deep Long Short-Term Memory (LSTM) recurrent neural network in a double-stream architecture. Such networks use memory cells in the hidden units, enabling them to learn long-range temporal context, and thus increasing the robustness against noise and reverberation. The network is trained to predict frame-wise phoneme estimates, which are converted into observation likelihoods to be used as an acoustic model. It is of particular interest whether the LSTM system is capable of improving a robust state-of-the-art GMM system, which is confirmed in the experimental results. In addition, we investigate the efficiency of NMF for speech enhancement on the front-end side. Experiments are conducted on the medium-vocabulary task of the 2nd ‘CHiME’ Speech Separation and Recognition Challenge, which includes reverberation and highly variable noise. Experimental results show that the average word error rate of the challenge baseline is reduced by 64% relative. The best challenge entry, a noise-robust state-of-the-art recognition system, is outperformed by 25% relative.

Learning Harmonium models with infinite latent features.

Undirected latent variable models represent an important class of graphical models that have been successfully developed to deal with various tasks. One common challenge in learning such models is to determine the number of hidden units that are unknown a priori. Although Bayesian nonparametrics have provided promising results in bypassing the model selection problem in learning directed Bayesian Networks, very little effort has been made toward applying Bayesian nonparametrics to learn undirected latent variable models. In this paper, we present the infinite exponential family Harmonium (iEFH), a bipartite undirected latent variable model that automatically determines the number of latent units from an unbounded pool. We also present two important extensions of iEFH to 1) multiview iEFH for dealing with heterogeneous data, and 2) infinite maximum-margin Harmonium (iMMH) for incorporating supervising side information to learn predictive latent features. We develop variational inference algorithms to learn model parameters. Our methods are computationally competitive because of the avoidance of selecting the number of latent units. Our extensive experiments on real image datasets and text datasets appear to demonstrate the benefits of iEFH and iMMH inherited from Bayesian nonparametrics and max-margin learning. Such results were not available until now and contribute to expanding the scope of Bayesian nonparametrics to learn the structures of undirected latent variable models.

Emulation of Spline Networks Through Approximation of Polynomials and Step Function of Neural Networks with Cosine Modulated Symmetric Exponential Function

This study proves that the neural network, using the cosine modulated symmetric exponential function which is a non-monotonic function, can emulate the spline networks by approximating the polynomials and step functions. This means that the network with cosine modulated symmetric exponential function is equivalent to the spline networks. It is also equivalent to the neural network, which uses sigmoidal, hyperbolic tangent, and Gaussian activation function, as proved by DasGupta and Schnitger. In the multi-network structure, the cosine modulated symmetric exponential function has the capability to make more local hills than any other functions. In the neural network that uses this function, it has the capability to quickly localize the input space pattern, even though it makes a fewer number of layers. On the other hand, the monotonic function needs a greater number of layers to make these local hills. Therefore, in the training for the pattern classification of the neural network, we need a greater number of units and epochs. This is connected to the training speed of the neural network for the pattern classification, which also indicates the capabilities of the network. For the capacity test of the pattern classification in the cosine modulated symmetric exponential function, we have used the Cascade-Correlation neural network. Cascade-Correlation is a supervised learning algorithm that automatically determines the size and topology of the network. The Cascade-Correlation adds new hidden units one by one and creates a multi-layer structure in which each unit is in a hidden layer. In this experiment, the two benchmark problems have been used: one is the iris plant classification problem; the other is the tic-tac-toe endgame problem. The results are compared with those obtained with other activation functions. In this experiment, the evaluation items, such as the number of epochs, produced hidden units, listing of the run time, and the average crossings per second of ten trials on the training set of the problem, have been compared. For instance, In the experiment of the iris plants classification problem, the CosExp function has recorded about 53 % of the average epochs number when compared to the sigmoid function, and for the number of the hidden units, it is approximately 54 %. In the tic-tac-toe problem experiment, the average number of epochs and hidden units produced during the process has been reduced approximately by one thirds more with the CosExp function than with other activation functions. Accordingly, learning has been improved three times faster. The results of the experiments show that performance can be improved very significantly by using the cosine modulated symmetric exponential function as the activation function in neural networks with a predetermined set of parameters.

Statistical and incremental methods for neural models selection

This work presents two methods of selection of neural models for identification of dynamic systems. Initially, a strategy of selection based on statistical tests, which relates to training and generalisation performances of a neural model is analysed. In the second time, a new constructive approach of neural model selection, which the training begins with minimal structure and then incrementally adds new hidden units and/or layers, is described. The simulation and the application of these methods for selection of neural models are also considered.

Design Neural Network to Solve Singular Perturbation Problems

The aim of this paper is to design neural network to present a method to solve Singular perturbation problems (SPP) by using network having one hidden layer with 5 hidden units (neurons) and one linear output unit, the sigmoid activation of each hidden units is tansigmoid. The neural network trained by the back propagation with different algorithms such as quasi-Newton, Levenberg-Marquardt, and Bayesian Regulation. Finally the results of numerical experiments are compared with the exact solution in illustrative examples to confirm the accuracy and efficiency of the presented scheme.

Research on Algorithm Optimization of Hidden Units Data Centre of RBF Neural Network

Common algorithms of selecting hidden unit data center in RBF neural networks were first discussed in this essay, i.e. k-means algorithm, subtractive clustering algorithm and orthogonal least squares. Meanwhile, a hybrid algorithm mixed of k-means algorithm and particle swarm optimization algorithm was put forward. The algorithm used the position of the particles in particle swarm optimization algorithm to help deal with the defects of local clusters resulted from k-means algorithm and to make optimization with the optimal fitness of k-means particle swarm with the aim to make the final optimal fitness better satisfy the requirements.

Learning Facial Expression Codes with Sparse Auto-Encoder

A sparse auto-encoder model was trained to extract the code of different facial expression, which comprises four encoder layers and three decode layers, the representation locating in the fourth layer (code layer) is the features expected. With large amounts of patches randomly selected from training faces, the model was trained firstly via backpropagation which minimizes an unsupervised sparse reconstruction error, and then a softmax classifier was learned for supervised classification. The input vector for the classification is the feature of facial image induced by the learned sparse auto-encoder and two key operations (convolving and pooling). Using a small number of hidden units per layer and a relatively small number of training set, the proposed model achieves excellent performance in the experiments.

Diminished acquired equivalence yet good discrimination performance in older participants

We asked younger and older human participants to perform computer-based configural discriminations that were designed to detect acquired equivalence. Both groups solved the discriminations but only the younger participants demonstrated acquired equivalence. The discriminations involved learning the preferences [“like” (+) or “dislike” (−)] for sports [e.g., tennis (t) and hockey (h)] of four fictitious people [e.g., Alice (A), Beth (B), Charlotte (C), and Dorothy (D)]. In one experiment, the discrimination had the form: At+, Bt−, Ct+, Dt−, Ah−, Bh+, Ch−, Dh+. Notice that, e.g., Alice and Charlotte are “equivalent” in liking tennis but disliking hockey. Acquired equivalence was assessed in ancillary components of the discrimination (e.g., by looking at the subsequent rate of “whole” versus “partial” reversal learning). Acquired equivalence is anticipated by a network whose hidden units are shared when inputs (e.g., A and C) signal the same outcome (e.g., +) when accompanied by the same input (t). One interpretation of these results is that there are age-related differences in the mechanisms of configural acquired equivalence.

Misclassifying New York’s Hidden Units as Vacant in 2010: Lessons Gleaned for the 2020 Census

The Local Update of Census Addresses (LUCA) program allowed local governments to include hard-to-find units in the Census Bureau’s Master Address File (MAF), which is the cornerstone of the mailout/mailback decennial census. These improvements have allowed the Census Bureau to penetrate the more marginal parts of the housing stock, where units are often not formally labeled, and where their very existence can be difficult to determine. In New York City, where address updating included two rounds of LUCA, the Census Bureau acknowledged an increase of 170,000 housing units between 2000 and 2010. However, there was a dramatic growth in vacant units, equivalent to almost one-half of the total increase in housing units. The increase in vacant units was disproportionately concentrated in 2 of the 18 local census offices in New York City. The paper uses local administrative data on new construction, property foreclosures, and property values; data from the United States Postal Service; as well as survey data from the New York City Housing and Vacancy Survey and the American Community Survey to show why this concentrated increase in vacant units is untenable. From the standpoint of the enumeration, units added in LUCA would challenge the best enumerator, but these hurdles were largely overcome, but for the two local census offices. The paper goes on to discuss how the Census Bureau can adopt measures in 2020 to ensure that housing units and their occupancy status are accurately enumerated in New York and across the nation.

Pattern separation by neuronal turnover in a feed-forward network

Numerous studies have revealed a role for the hippocampal dentate gyrus in behavioral discrimination between similar contexts or objects, referred to as pattern separation. Recently many behavioral studies have demonstrated a role for dentate neurogenesis in such pattern separation. While several computational studies have modeled the effect of neuronal turnover on learning in simple [1-3] and more detailed networks [4], the computational advantage of neurogenesis for pattern separation has remained obscure. Here we present a simple, feed-forward network, with a biologically plausible learning rule for implementing neuronal recycling that reveals the pattern separation properties of neurogenesis. Our model consists of a three-layer network including entorhinal cortical inputs (EC), dentate gyrus (DG) as the hidden layer, and a single CA3 readout unit. The weights between EC and DG are fixed random, while those from DG to CA3 are trained with a perceptron learning rule. A context consists of a set of patterns presented to EC with their respective target readouts at CA3. Pereceptron training results in a weight vector with approximately normally distributed weights. Neurogenesis is implemented by replacing units with weak weights to the CA3 readout, thus a new hidden unit with random weights replaces the prior unit. Thereupon another round of perceptron learning is implemented. This simple rule results in a markedly reduced generalization error at the CA3 readout that continues to decline with each subsequent round of neurogenesis. Intuitively, this neurogenesis improves linear separability in the DG space allowing the CA3 readout to draw a decision hyperplane with a greater margin between the classes. This form of neuronal turnover may represent a biologically plausible replacement for error backpropagation.

Combination of supervised and unsupervised learning for training the activation functions of neural networks

Standard feedforward neural networks benefit from the nice theoretical properties of mixtures of sigmoid activation functions, but they may fail in several practical learning tasks. These tasks would be better faced by relying on a more appropriate, problem-specific basis of activation functions. The paper presents a connectionist model which exploits adaptive activation functions. Each hidden unit in the network is associated with a specific pair (f(·),p(·)), where f(·) is the activation function and p(·) is the likelihood of the unit being relevant to the computation of the network output over the current input. The function f(·) is optimized in a supervised manner, while p(·) is realized via a statistical parametric model learned through unsupervised (or, partially supervised) estimation. Since f(·) and p(·) influence each other’s learning process, the overall machine is implicitly a co-trained coupled model and, in turn, a flexible, non-standard neural architecture. Feasibility of the approach is corroborated by empirical evidence yielded by computer simulations involving regression and classification tasks.