Pulsed Neural Networks Research Articles

BLOWOUT BIFURCATION AND ON–OFF INTERMITTENCY IN PULSE NEURAL NETWORKS WITH MULTIPLEC MODULES

To study the mechanism by which high-dimensional chaos emerges in neural systems, the synchronization of chaotic firings in class 1 pulse neural networks composed of excitatory and inhibitory ensembles was analyzed. In the system with two modules (i.e. two pulse neural networks), blowout bifurcation and on–off intermittency were observed when the inter-module connection strengths were reduced from large values. In the system with three modules, rearrangement of synchronized clusters and chaotic itinerancy were observed. Such dynamics may be one of the mechanisms through which high-dimensional chaos is generated in neural systems.

Analysis of Synchronization Between Two Modules of Pulse Neural Networks with Excitatory and Inhibitory Connections

To study the synchronized oscillations among distant neurons in the visual cortex, we analyzed the synchronization between two modules of pulse neural networks using the phase response function. It was found that the intermodule connections from excitatory to excitatory ensembles tend to stabilize the antiphase synchronization and that the intermodule connections from excitatory to inhibitory ensembles tend to stabilize the in-phase synchronization. It was also found that the intermodule synchronization was more noticeable when the inner-module synchronization was weak.

Analysis of Synchronization Between Two Modules of Pulse Neural Networks with Excitatory and Inhibitory Connections

To study the synchronized oscillations among distant neurons in the visual cortex, we analyzed the synchronization between two modules of pulse neural networks using the phase response function. It was found that the intermodule connections from excitatory to excitatory ensembles tend to stabilize the antiphase synchronization and that the intermodule connections from excitatory to inhibitory ensembles tend to stabilize the in-phase synchronization. It was also found that the intermodule synchronization was more noticeable when the inner-module synchronization was weak.

Stability estimation of PFM-type pulsed neural networks

This study aims at development of practical stability estimation method for PFM-type pulsed neural networks. We already derived a graphical analysis method for a single-unit PFM-type neural network. In this paper, we extend this method so that it can deal with stability estimation of networks with multi-unit PFM-type pulsed neurons by using the concept of Generalised Nyquist Stability Criterion. We also illustrate numerical examples that show the extended method is not excessively conservative but still effectively estimates the non-existence of limit cycles.

Pulsed neural networks based on delta‐sigma modulation with GHA learning rule and their hardware implementation

AbstractIn order to achieve fast parallel processing operation by a neural network, it is desirable to implement all neurons in parallel in hardware rather than to realize operation by serial operation in software. The authors have proposed a pulsed neural network based on ΔΣ modulation which is suited to the hardware implementation of digital circuits. The proposed neural network can be realized on a small circuit scale, since the variable can be represented by a ΔΣ‐modulated 1‐bit pulse signal. Highly accurate operation can be realized even though only 1 bit is used. This paper proposes a hardware implementation of the above neural network. A neural network with built‐in GHA learning rule is hardware implemented on an FPGA, and its operation is investigated. The proposed neural network and CPU are implemented on an FPGA and are compared in terms of the number of bits that can be processed in unit time per unit of circuit scale. The proposed scheme achieved an evaluation score 200 times as great as a software realization on a CPU. © 2005 Wiley Periodicals, Inc. Syst Comp Jpn, 36(9): 14–24, 2005; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.20235

A pulse neural network incorporating short term synaptic plasticity for engineering applications

A pulse neural network incorporating short term synaptic plasticity for engineering applications

Hebbian learning rule restraining catastrophic forgetting in pulse neural network

AbstractIn this paper, a Hebbian learning rule restraining “catastrophic forgetting” is proposed on a pulsed neural network (PNN) with leaky integrate‐and‐fire neurons. The strong point of this learning rule is that a learning of new pattern does not destroy past ones, and that an efficient use of synapses is enabled. First, in order to consider the function of the learning rule, a fundamental experiment is carried out. Next, to compare the performance between the proposed learning rule and conventional ones on the application, simulation experiments are examined using autonomous behavior robots which are forced to learn concurrently two different environments. The results of the experiments show that the proposed learning rule clearly restrains “catastrophic forgetting” and enables working of more efficient than conventional PNN learning. © 2005 Wiley Periodicals, Inc. Electr Eng Jpn, 151(3): 50–60, 2005; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/eej.10343

A pulse neural network reinforcement learning algorithm for partially observable Markov decision processes

AbstractThis paper considers learning by a pulse neural network and proposes a new reinforcement learning algorithm focusing on the ability of pulse neuron elements to process time series. The conventional integrator neuron element is modeled in terms of the average firing rate of the biological neuron. But the pulse neuron is a modeling of the input–output relation of the time‐series pulse (spike) and the decay of the internal state (internal potential). The application of such neural networks has been considered in recent engineering studies. It is known in particular that a pulse neuron with a high decay rate acts as a coincidence detector. The proposed model combines pulse neuron elements with different decay rates, which facilitates the processing of the time‐series input information and the discrimination of fuzzy states in a partially observable Markov decision process. The proposed network is a four‐layered feedforward network in which the pulse neuron elements forming the two hidden layers provide a pseudo‐representation of the state in the environment. The elements generate a secondary reinforcement signal which results in learning similar to the conventional reinforcement scheme based on the state evaluation function. A computer experiment verifies that the proposed model works effectively in an environment which is strongly partially observable. © 2005 Wiley Periodicals, Inc. Syst Comp Jpn, 36(3): 42–52, 2005; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10645

Visual Attention Control using Pulse Neural Network with Short Term Synaptic Depression

In this paper, we propose a new artificial pulse neural network model which utilizes short term synaptic depression (STSD) for attention control. Compared with classic integrator-type neurons, pulse neurons are known for their superior ability to handle temporal information, as well as the easiness to implement new found properties of biological neurons. STSD is one of such properties. While recent researches indicate that STSD shows some interesting behaviours like gain controls, its usefulness for artificial neural networks is still unclear. The proposed model is a brand new pulse neural network to utilize STSD for movie processing. This model is composed of four layers of pulse neurons. The input is the brightness information of the current image of the movie, and the output is the area which requires system's attention. Due to STSD between the input layer and the first hidden layer, it is capable of keeping its attention on things like fast moving objects, while ignoring static or slow objects even if they are flickering. Experimental results show that this model has the basic abilities for attention control.

A pulsed neural network model of spectro-temporal receptive fields and population coding in auditory cortex

The existence of spectro-temporal receptive fields and evidence for population coding in auditory cortex motivate the development of such models, that explicitly operate in the time-frequency domain and are based on a pulsed neural network. In presenting such a model, a formal connection of the fields of Time Frequency Analysis and Pulsed Neural Networks is established. The resulting neural time-frequency signal representation is shown to be representable as a signal-dependent overcomplete dictionary. It is derived from neural population coding. Signal decomposition and filtering effects are presented, indicating obvious technical applications of the proposed model.

ＭＩマイクロセンサアレイを用いた可変シナプス結合素子による磁気帰還型ニューロ回路

This paper investigates a magnetic-feedback-type neural circuit based on a variable-synapse coupling device using an MI microsensor array. The proposed variable-synapse device is a magnetic coupling system for multiplication and accumulation. Analogue magnetic memory will be utilized to store the values of synapse couplings. The pulse modulation characteristics of the magnetic-feedback-type neural circuit are demonstrated. A pulse neural network system could be devised, using the magnetic synapse in combination with a pulse density modulation circuit.

A Discrete-Event Neural Network Simulator for General Neuron Models

Efficient simulation techniques for a discrete-event pulsed neural network simulator are developed. In a discrete-event simulation framework, simulation of complex neural behaviours, such as phase precession and phase arbitration, demands the prediction of delayed firing times. The new technique, the incremental partitioning method, uses linear envelopes of the state variable of a neuron to partition the simulated time so that the delayed-firing time is reliably calculated by applying the bisection-combined Newton-Raphson method to every partition. The quick filtering technique is also proposed for reducing calculation cost of linear envelopes. The simulator developed, Punnets, has achieved efficiency and precision, but is still capable of simulating a complex behaviour of large-scale neural network models.

Hebbian Learning Rule Restraining Catastrophic Forgetting in Pulse Neural Network

In this paper, a Hebbian learning rule restraining “catastrophic forgetting” is proposed on a pulsed neural network (PNN) with leaky integrate-and-fire neurons. The strong point of this learning rule is that a learning of new pattern does not destroy past ones, and that an efficient use of synapses is enabled. First, in order to consider the function of the learning rule, a fundamental experiment is carried out. Next, to compare the performance between the proposed learning rule and conventional ones on the application, simulation experiments are examined using autonomous behavior robots which are forced to learn concurrently two different environments. The results of the experiments show that the proposed learning rule clearly restrains “catastrophic forgetting” and enables working of more efficient than conventional PNN learning. © 2005 Wiley Periodicals, Inc. Electr Eng Jpn, 151(3): 50–60, 2005; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/eej.10343

Imperfect pulse detection in neural nets with temporal codes

The statistics of pulsed neural networks employing temporal codes (time to rth pulse) are investigated under conditions of imperfect pulse detection. The mixed negative binomial distributions that result are broadened by the errors without changing their essential shapes. For r=1, the near-geometric distributions primarily exhibit decreasing effective rate parameters. These results are important in deciding between rate codes and temporal codes in noisy or otherwise imperfect artificial neural (or biological) environments.

1A1-E03 パルスニューラルネットワークのハードウェア進化に関する一考察

1A1-E03 パルスニューラルネットワークのハードウェア進化に関する一考察

Dynamic range and error tolerance of stochastic neural rate codes

An investigation is made of the statistical properties of unary stochastic rate codes, as employed in the implementation of pulsed neural networks. The dynamic range exhibits a reduced dependence on the representation time due to increased estimation errors from the variance in the stochastic encodings of the integers. At the same time, these codes exhibit improved tolerance to noise-induced bit errors. Comparisons are made with conventional integer arithmetic and with deterministic unary codes for neural networks, up to bit error rates as large as 0.3.

Fluctuation-induced memory retrieval in a pulsed neural network storing sparse patterns with hierarchical correlations.

An associative memory in a pulsed neural network composed of the FitzHugh-Nagumo models storing sparse patterns with hierarchical correlations is investigated. The memory patterns composed of 0/1 digits are represented by the synchronous periodic firings of the neurons. It is found that the target pattern and the OR pattern are retrieved individually by controlling the intensity of fluctuations in the system.

Dynamics of stochastic artificial neurons

Up–down saturating counters driven by stochastic signals (Bernoulli processes) are used to implement stochastic neurons in digital pulsed neural networks. The transfer functions of these neurons relate the probabilities of the output and input pulses and can approximate either sigmoidal or gaussian activation functions. The output is in general a simple combinational logic function of the various bits of the b-bit counter. The dynamics of the state transitions are presented in this work. The times required for the state transitions to converge to equilibrium (stationary states) depend upon the input Bernoulli probability p in a complex manner. The sigmoidal or gaussian approximations improve with the number of states N=2 b of the counter, and convergence times increase approximately as ( N−1) 2 as expected for a biased random walk or diffusion process.

A pulsed neural network model of bursting in the basal ganglia

We present new techniques for extending the functionality of spiking neurons which allow the incorporation of several aspects of neuron function previously confined to the domain of low level ion-channel based models. These aspects include spontaneous (or endogenous) firing, the complex interaction of multiple ion-species and the spatial distribution of synaptic contacts over the cell membrane. These ideas are applied to a neural circuit consisting of the cortex and a subset of the nuclei in the basal ganglia—the subthalamic nucleus (STN) and the external segment of the globus pallidus (GPe). This circuit has been studied extensively in vitro by Plenz and Kitai [Plenz, D., & Kitai, S. T. (1999). A basal ganglia pacemaker formed by the subthalamic nucleus and external globus pallidus. Nature, 400 677–682] whose data we use to constrain our model. With respect to this circuit, we have obtained three main results. First, that its characteristic burst firing is due to a Ca 2+ current mediated mechanism. Second, that noise can assist in the generation of bursting and, paradoxically, stabilise the network behaviour under synaptic weight variations. Third, that a variety of dendritic processing is necessary in order to obtain the full range of bursting behaviour.

On the relevance of time in neural computation and learning

We discuss models for computation in biological neural systems that are based on the current state of knowledge in neurophysiology. Differences and similarities to traditional neural network models are highlighted. It turns out that many important questions regarding computation and learning in biological neural systems cannot be adequately addressed in traditional neural network models. In particular, the role of time is quite different in biologically more realistic models, and many fundamental questions regarding computation and learning have to be rethought for this context. Simultaneously, a somewhat related new generation of VLSI-chips is emerging (“pulsed VLSI”) where new ideas about computing and learning with temporal coding can be tested in an engineering context. Articles with details to models and results that are sketched in this article can be found at http: //www.tu-graz.ac.at /igi /maass /. We refer to Maass and Bishop (Eds., Pulsed Neural Network, MIT Press, Cambridge, MA, 1999) for a collection of survey articles that contain further details and references.