Weight Pruning Research Articles

A modular attractor associative memory with patchy connectivity and weight pruning

An important research topic in neuroscience is the study of mechanisms underlying memory and the estimation of the information capacity of the biological system. In this report we investigate the performance of a modular attractor network with recurrent connections similar to the cortical long-range connections extending in the horizontal direction. We considered a single learning rule, the BCPNN, which implements a kind of Hebbian learning and we trained the network with sparse random patterns. The storage capacity was measured experimentally for networks of size between 500 and 46 K units with a constant activity level, gradually diluting the connectivity. We show that the storage capacity of the modular network with patchy connectivity is comparable with the theoretical values estimated for simple associative memories and furthermore we introduce a new technique to prune the connectivity, which enhances the storage capacity up to the asymptotic value.

A novel weight pruning method for MLP classifiers based on the MAXCORE principle

We introduce a novel weight pruning methodology for MLP classifiers that can be used for model and/or feature selection purposes. The main concept underlying the proposed method is the MAXCORE principle, which is based on the observation that relevant synaptic weights tend to generate higher correlations between error signals associated with the neurons of a given layer and the error signals propagated back to the previous layer. Nonrelevant (i.e. prunable) weights tend to generate smaller correlations. Using the MAXCORE as a guiding principle, we perform a cross-correlation analysis of the error signals at successive layers. Weights for which the cross-correlations are smaller than a user-defined error tolerance are gradually discarded. Computer simulations using synthetic and real-world data sets show that the proposed method performs consistently better than standard pruning techniques, with much lower computational costs.

Generation, elimination and weight fluctuations of synapses in the cerebral cortex

Most currently available neuronal network models define learning as changes in the strength of synaptic connectivity (i.e. weight plasticity), with synaptic weights assumed to remain constant in the absence of such learning. However, our recent study on excitatory neuronal networks in the cerebral cortex has revealed that, in addition to activity-dependent plasticity, these networks exhibit intrinsic fluctuations in synaptic weight and widespread spontaneous synapse formation and pruning, with a timecourse on the order of days. Our results indicate a coupling between weight plasticity and spine generation/elimination, and suggest that these mechanisms have complimentary but distinct functions in memory formation and maintenance. Three principles of plasticity in spine synapses are presented here to facilitate construction of a new network model for the cerebral cortex.

Simple conditions for forming triangular grids

We have used simple learning rules to study how firing maps containing triangular grids—as found in in vivo experiments—can be developed by Hebbian means in realistic robotic simulations. We started from typical non-local postrhinal neuronal responses. We found that anti-Hebbian weight pruning can develop triangular grids under certain conditions. Experimental evidences and the present study suggest that within this model, whitening is a bottom-up process, whereas weight pruning and possibly the non-linear extension of whitening to bottom-up information maximization are guided by top-down influences that reorganize entorhinal responses. We connect our model to the computational model of the entorhinal–hippocampal region of Lőrincz and Buzsáki. In the joined model, the hippocampus is the origin of response reorganization. The joined model may provide insights for memory reorganization guided by hippocampal supervision.

NOVEL ADAPTIVE FILTERING TECHNIQUE FOR THE PROCESSING OF ELECTROGASTROGRAM

The electrogastrogram measured cutaneously by attaching electrodes to the abdominal skin contains considerable noise. An attempt is made to develop an efficient adaptive filtering technique suitable for real time processing of electrogastrogram. A novel technique combining both adaptive noise cancellation and adaptive signal enhancement in a single recurrent neural network is proposed. To compare its performance, adaptive noise cancellation and cascaded connection of adaptive noise cancellation and adaptive signal enhancement are employed. Recurrent neural networks using Real Time Recurrent Learning (RTRL) algorithm are employed for implementing the all the above systems. An attempt is made to alleviate the computational burden imposed by RTRL algorithm by employing pruning of weights.

Recognition of environmental and genetic effects on barley phenolic fingerprints by neural networks

Through computational analysis of high-performance liquid chromatography (HPLC) traces we find correlations between secondary metabolites and growth conditions of six varieties of barley. Using artificial neural networks, it was possible to classify chromatograms for which the varieties were fertilized by nitrogen and treated by fungicide. For each variety of barley we could also differentiate it from the others. Surprisingly, all these classification tasks could be solved successfully by a simple network with no hidden units. When adding to the methodology pruning of the network weights, we were able to reduce the set of peaks in the chromatograms and obtain a necessary subset from which the growth conditions and differentiation may be decided. In some instances, more complex networks with hidden units could lead to a further reduction of the number of peaks used. In most cases, far more than half of the peaks are redundant. We find that it requires fewer information-rich peaks to perform the variety differentiation tasks than to recognize any of the growth conditions. Analysis of the network weights reveals correlations between weighted combinations of peaks.

A formal selection and pruning algorithm for feedforward artificial neural network optimization

A formal selection and pruning technique based on the concept of local relative sensitivity index is proposed for feedforward artificial neural networks. The mechanism of backpropagation training algorithm is revisited and the theoretical foundation of the improved selection and pruning technique is presented. This technique is based on parallel pruning of weights which are relatively redundant in a subgroup of a feedforward neural network. Comparative studies with a similar technique proposed in the literature show that the improved technique provides better pruning results in terms of reduction of model residues, improvement of generalization capability and reduction of network complexity. The effectiveness of the improved technique is demonstrated in developing neural network (NN) models of a number of nonlinear systems including three bit parity problem, Van der Pol equation, a chemical processes and two nonlinear discrete-time systems using the backpropagation training algorithm with adaptive learning rate.

Exploiting domain-specific properties: compiling parallel dynamic neural network algorithms into efficient code

Domain-specific constraints can be exploited to implement compiler optimizations that are not otherwise feasible. Compilers for neural network learning algorithms can achieve near-optimal colocality of data and processes and near-optimal balancing of load over processors, even for dynamically irregular problems. This is impossible for general programs, but restricting programs to the neural algorithm domain allows for the exploitation of domain-specific properties. The operations performed by neural algorithms are broadcasts, reductions, and object-local operations only; the load distribution is regular with respect to the (perhaps irregular) network topology; changes of network topology occur only from time to time. A language, compilation techniques, and a compiler implementation on the MasPar MP-1 are described and quantitative results for the effects of various optimizations used in the compiler are shown. Conservative experiments with weight pruning algorithms yield performance improvements of 27 percent due to load balancing and 195 percent improvement is achieved due to data locality, both compared to unoptimized versions. Two other optimizations-connection allocation and selecting the number of replicates-speed programs up by about 50 percent and 100 percent, respectively. This work can be viewed as a case study in exploiting domain-specific information; some of the principles presented here may apply to other domains as well.

Stabilization and speedup of convergence in training feedforward neural networks

We review the training problem for feedforward neural networks and discuss various techniques for accelerating and stabilizing the convergence during training. Among other techniques, these include a self-adjusting step gain, bipolar sigmoid activation functions, training on all classes in parallel, adjusting the exponential rates in the sigmoids, bounding the sigmoid derivatives away from zero, training on exemplars to which noise has been added, adjusting the initial weight set to a subdomain of low values of the sum-squared error, and adjusting the momentum coefficient over the iterations. We also examine methods to assure the generalization of the learning, which include the pruning of unimportant weights and adding noise to exemplars for training.

Two original weight pruning methods based on statistical tests and rounding techniques

The authors focus on the use of neural networks to approximate continuous decision functions. In this context, the parameters to be estimated are the synaptic weights of the network. The number of such parameters and the quantity of data (information) available for training greatly influence the quality of the solution obtained. A previous study analysed the influence and interaction of these two features. In order to reach the architecture of the net leading to the best fitting of the training data, two original pruning techniques are proposed. The evolution of the neural network performances, training and test rates, as the number of synaptic weights pruned increases, is shown experimentally. Two kinds of synaptic weights are obvious: irrelevant synaptic weights, which can be suppressed from the model; and relevant synaptic weights, which cannot be removed. In the test problem, it is possible to reduce the size of the network up to 42%. A 4% improvement of the performance in generalisation is observed.

Improving model selection by nonconvergent methods

Many techniques for model selection in the field of neural networks correspond to well established statistical methods. For example, architecture modifications based on test variables calculated after convergence of the training process can be viewed as part of a hypothesis testing search, and the use of complexity penalty terms is essentially a type of regularization or biased regression. The method of “stopped” or “cross-validation” training, on the other hand, in which an oversized network is trained until the error on a further validation set of examples deteriorates, then training is stopped, is a true innovation since model selection doesn't require convergence of the training process. Here, the training process is used to perform a directed search of the parameter space for a model which doesn't overfit the data and thus demonstrates superior generalization performance. In this paper we show that this performance can be significantly enhanced by expanding the “nonconvergent method” of stopped training to include dynamic topology modifications (dynamic weight pruning) and modified complexity penalty term methods in which the weighting of the penalty term is adjusted during the training process. On an extensive sequence of simulation examples we demonstrate the general superiority of the “extended” nonconvergent methods compared to classical penalty term methods, simple stopped training, and methods which only vary the number of hidden units.