Differential Hebbian Learning Research Articles

Differential Hebbian learning with time-continuous signals for active noise reduction.

Spike timing-dependent plasticity, related to differential Hebb-rules, has become a leading paradigm in neuronal learning, because weights can grow or shrink depending on the timing of pre- and post-synaptic signals. Here we use this paradigm to reduce unwanted (acoustic) noise. Our system relies on heterosynaptic differential Hebbian learning and we show that it can efficiently eliminate noise by up to -140 dB in multi-microphone setups under various conditions. The system quickly learns, most often within a few seconds, and it is robust with respect to different geometrical microphone configurations, too. Hence, this theoretical study demonstrates that it is possible to successfully transfer differential Hebbian learning, derived from the neurosciences, into a technical domain.

A differential Hebbian framework for biologically-plausible motor control

In this paper we explore a neural control architecture that is both biologically plausible, and capable of fully autonomous learning. It consists of feedback controllers that learn to achieve a desired state by selecting the errors that should drive them. This selection happens through a family of differential Hebbian learning rules that, through interaction with the environment, can learn to control systems where the error responds monotonically to the control signal. We next show that in a more general case, neural reinforcement learning can be coupled with a feedback controller to reduce errors that arise non-monotonically from the control signal. The use of feedback control can reduce the complexity of the reinforcement learning problem, because only a desired value must be learned, with the controller handling the details of how it is reached. This makes the function to be learned simpler, potentially allowing learning of more complex actions. We use simple examples to illustrate our approach, and discuss how it could be extended to hierarchical architectures.

A Data and Knowledge Collaboration Strategy for Decision-Making on the Amount of Aluminum Fluoride Addition Based on Augmented Fuzzy Cognitive Maps

Abstract In the aluminum reduction process, aluminum fluoride (AlF3) is added to lower the liquidus temperature of the electrolyte and increase the electrolytic efficiency. Making the decision on the amount of AlF3 addition (referred to in this work as MDAAA) is a complex and knowledge-based task that must take into consideration a variety of interrelated functions; in practice, this decision-making step is performed manually. Due to technician subjectivity and the complexity of the aluminum reduction cell, it is difficult to guarantee the accuracy of MDAAA based on knowledge-driven or data-driven methods alone. Existing strategies for MDAAA have difficulty covering these complex causalities. In this work, a data and knowledge collaboration strategy for MDAAA based on augmented fuzzy cognitive maps (FCMs) is proposed. In the proposed strategy, the fuzzy rules are extracted by extended fuzzy k-means (EFKM) and fuzzy decision trees, which are used to amend the initial structure provided by experts. The state transition algorithm (STA) is introduced to detect weight matrices that lead the FCMs to desired steady states. This study then experimentally compares the proposed strategy with some existing research. The results of the comparison show that the speed of FCMs convergence into a stable region based on the STA using the proposed strategy is faster than when using the differential Hebbian learning (DHL), particle swarm optimization (PSO), or genetic algorithm (GA) strategies. In addition, the accuracy of MDAAA based on the proposed method is better than those based on other methods. Accordingly, this paper provides a feasible and effective strategy for MDAAA.

General differential Hebbian learning: Capturing temporal relations between events in neural networks and the brain.

Learning in biologically relevant neural-network models usually relies on Hebb learning rules. The typical implementations of these rules change the synaptic strength on the basis of the co-occurrence of the neural events taking place at a certain time in the pre- and post-synaptic neurons. Differential Hebbian learning (DHL) rules, instead, are able to update the synapse by taking into account the temporal relation, captured with derivatives, between the neural events happening in the recent past. The few DHL rules proposed so far can update the synaptic weights only in few ways: this is a limitation for the study of dynamical neurons and neural-network models. Moreover, empirical evidence on brain spike-timing-dependent plasticity (STDP) shows that different neurons express a surprisingly rich repertoire of different learning processes going far beyond existing DHL rules. This opens up a second problem of how capturing such processes with DHL rules. Here we propose a general DHL (G-DHL) rule generating the existing rules and many others. The rule has a high expressiveness as it combines in different ways the pre- and post-synaptic neuron signals and derivatives. The rule flexibility is shown by applying it to various signals of artificial neurons and by fitting several different STDP experimental data sets. To these purposes, we propose techniques to pre-process the neural signals and capture the temporal relations between the neural events of interest. We also propose a procedure to automatically identify the rule components and parameters that best fit different STDP data sets, and show how the identified components might be used to heuristically guide the search of the biophysical mechanisms underlying STDP. Overall, the results show that the G-DHL rule represents a useful means to study time-sensitive learning processes in both artificial neural networks and brain.

Uncertainty Propagation in Fuzzy Grey Cognitive Maps With Hebbian-Like Learning Algorithms

This paper is focused on an innovative fuzzy cognitive maps extension called fuzzy grey cognitive maps (FGCMs). FGCMs are a mixture of fuzzy cognitive maps and grey systems theory. These have become a useful framework for facing problems with high uncertainty, under discrete small and incomplete datasets. This paper deals with the problem of uncertainty propagation in FGCM dynamics with Hebbian learning. In addition, this paper applies differential Hebbian learning (DHL) and balanced DHL to FGCMs for the first time. We analyze the uncertainty propagation in eight different scenarios in a classical chemical control problem. The results give insight into the propagation of the uncertainty or greyness in the iterations of the FGCMs. The results show that the nonlinear Hebbian learning is the choice with less uncertainty in steady final grey states for Hebbian learning algorithms.

Predictive Acoustic Tracking with an Adaptive Neural Mechanism

Tracking an acoustic signal in motion is pertinent in several domains such as human-robot interaction and search-and-rescue robotics. Conventional approaches to acoustic tracking acquire time-of-arrival-difference signals from multi-microphone arrays and localise the acoustic signal using Kalman or particle filtering, generalised cross-correlation or steered response power techniques. The authors have previously developed a biologically-inspired mechanism that utilises two microphones to reactively track an acoustic signal in motion. The mechanism leverages the directional response of an mathematical model of the lizard peripheral auditory system to extract information regarding sound direction. This information is utilised by a neural machinery to learn the acoustic signal's velocity through fast and unsupervised correlation-based learning adapted from differential Hebbian learning. This approach has previously been validated in simulation and via robotic trials to track a continuous pure tone acoustic signal with a semi-circular motion trajectory and a constant but unknown angular velocity. The neural machinery has been shown to be able to learn different target angular velocities in independent trials. Here we extend our previous work by demonstrating that an identical instance of the mechanism can be used to successfully predict the future spatial location of an acoustic signal with an identical semi-circular motion trajectory and a constant but unknown angular velocity. We evaluate the prediction performance of the simulated mechanism in independent trials for three different angular velocities.

Fuzzy cognitive maps of public support for insurgency and terrorism

Feedback fuzzy cognitive maps (FCMs) can model the complex structure of public support for insurgency and terrorism (PSOT). FCMs are fuzzy causal signed digraphs that model degrees of causality in interwoven webs of feedback causality and policy variables. Their nonlinear dynamics permit forward-chaining inference from input causes and policy options to output effects. We show how a concept node causally affects downstream nodes through a weighted product of the intervening causal edge strengths. FCMs allow users to add detailed dynamics and feedback links directly to the causal model. Users can also fuse or combine FCMs from multiple experts by weighting and adding the underlying FCM fuzzy edge matrices. The combined FCM tends to better represent domain knowledge as the expert sample size increases if the expert sample approximates a random sample. Statistical or machine-learning algorithms can use numerical sample data to learn and tune a FCM’s causal edges. A differential Hebbian learning law can approximate a PSOT FCM’s directed edges of partial causality using time-series training data. The PSOT FCM adapts to the computational factor-tree PSOT model that Davis and OMahony based on prior social science research and case studies. Simulation experiments compare the PSOT models with the adapted FCM models.

In Search for the Neural Mechanisms of Individual Development: Behavior-Driven Differential Hebbian Learning

When Donald Hebb published his 1949 book ``The Organization of Behavior'' he opened a new way of thinking in theoretical neuroscience which, in retrospective, is very close to contemporary ideas in self-organization. His metaphor of ``wiring'' together what ``fires together'' matches very closely the common paradigm that global organization can derive from simple local rules. While ingenious at his time and inspiring the research over decades, the results still fall short of the expectations. For instance, unsupervised as they are, such neural mechanisms should be able to explain and realize the self-organized acquisition of sensorimotor competencies. This paper proposes a new synaptic law which replaces Hebb's original metaphor by that of ``chaining together'' what ``changes together''. Starting from differential Hebbian learning, the new rule grounds the behavior of the agent directly in the internal synaptic dynamics. Therefore, one may call this a behavior-driven synaptic plasticity. Neurorobotics is an ideal testing ground for this new, unsupervised learning rule. This paper focuses on the close coupling between body, control, and environment in challenging physical settings. The examples demonstrate how the new synaptic mechanism induces a self-determined ``search and converge'' strategy in behavior space, generating spontaneously a variety of sensorimotor competencies. The emerging behavior patterns are qualified by involving body and environment in an irreducible conjunction with the internal mechanism. The results may not only be of immediate interest for the further development of embodied intelligence. They also offer a new view on the role of self-learning processes in natural evolution and in the brain. Videos and further details may be found under \url{http://robot.informatik.uni-leipzig.de/research/supplementary/NeuroAutonomy/}.

Theta rhythmicity enhances learning in adaptive STDP

The classical STDP window captures changes of a synaptic weight in response to the relative timing of a pre and a postsynaptic spike (see e.g. Bi and Poo, 1998). Due to its static nature, however, it cannot account for nonlinear interactions between spikes. Several theoretical studies offer dynamic formulations for STDP, for example by modulating the synaptic weight change by variables like synaptic calcium concentration (Shouval et al., 2002) or somatic depolarisation (Clopath et al., 2010), or by introducing spike triplet interactions (Pfister and Gerstner, 2006). Here, we propose a new model which is formulated as a set of differential equations (Schmiedt et al., 2010). The weight change is given by a differential Hebbian learning rule, which reproduces the STDP window for spike pairs. To account for the effects of repeated neuronal firing on the synaptic weight, we introduce modulations of the spike impact, which act on exponential traces of the spiking activity. We found that this model captures a series of experiments on STDP with complex spike pattern in cortex (Froemke et al., 2006) and hippocampus (Wang et al., 2005). When applied to continuous firing rates, our approach allows us to analyze the effects of given time courses of firing rates on the synaptic weight change, i.e. the filter properties of STDP. For sinusoidal modulations of baseline firing rates we find the strongest weight changes for modulation frequencies in the theta band, which plays a key role in learning. Furthermore, weight modifications in the hippocampus are predicted to be most prominent for baseline rates of around 5Hz in striking agreement with experimental findings.This suggests that STDP-dependent learning is mediated by theta oscillations and modulated by the background firing rate which are both testable predictions of our theory.

How feedback inhibition shapes spike-timing-dependent plasticity and its implications for recent Schizophrenia models

It has been shown that plasticity is not a fixed property but, in fact, changes depending on the location of the synapse on the neuron and/or changes of biophysical parameters. Here, we investigate how plasticity is shaped by feedback inhibition in a cortical microcircuit. We use a differential Hebbian learning rule to model spike-timing-dependent plasticity and show analytically that the feedback inhibition shortens the time window for LTD during spike-timing-dependent plasticity but not for LTP. We then use a realistic GENESIS model to test two hypothesis about interneuron hypofunction and conclude that a reduction in GAD67 is the most likely candidate as the cause for hypofrontality as observed in Schizophrenia.

Behavioral analysis of differential hebbian learning in closed-loop systems

Understanding closed loop behavioral systems is a non-trivial problem, especially when they change during learning. Descriptions of closed loop systems in terms of information theory date back to the 1950s, however, there have been only a few attempts which take into account learning, mostly measuring information of inputs. In this study we analyze a specific type of closed loop system by looking at the input as well as the output space. For this, we investigate simulated agents that perform differential Hebbian learning (STDP). In the first part we show that analytical solutions can be found for the temporal development of such systems for relatively simple cases. In the second part of this study we try to answer the following question: How can we predict which system from a given class would be the best for a particular scenario? This question is addressed using energy, input/output ratio and entropy measures and investigating their development during learning. This way we can show that within well-specified scenarios there are indeed agents which are optimal with respect to their structure and adaptive properties.

Erratum to: Self-influencing synaptic plasticity: recurrent changes of synaptic weights can lead to specific functional properties

Recent experimental results suggest that dendritic and back-propagating spikes can influence synaptic plasticity in different ways (Holthoff, 2004; Holthoff et al., 2005). In this study we investigate how these signals could interact at dendrites in space and time leading to changing plasticity properties at local synapse clusters. Similar to a previous study (Saudargiene et al., 2004) we employ a differential Hebbian learning rule to emulate spike-timing dependent plasticity and investigate how the interaction of dendritic and back-propagating spikes, as the post-synaptic signals, could influence plasticity. Specifically, we will show that local synaptic plasticity driven by spatially confined dendritic spikes can lead to the emergence of synaptic clusters with different properties. If one of these clusters can drive the neuron into spiking, plasticity may change and the now arising global influence of a back-propagating spike can lead to a further segregation of the clusters and possibly the dying-off of some of them leading to more functional specificity. These results suggest that through plasticity being a spatial and temporal local process, the computational properties of dendrites or complete neurons can be substantially augmented.

Closed-Form Treatment of the Interactions between Neuronal Activity and Timing-Dependent Plasticity in Networks of Linear Neurons

Network activity and network connectivity mutually influence each other. Especially for fast processes, like spike-timing-dependent plasticity (STDP), which depends on the interaction of few (two) signals, the question arises how these interactions are continuously altering the behavior and structure of the network. To address this question a time-continuous treatment of plasticity is required. However, this is - even in simple recurrent network structures - currently not possible. Thus, here we develop for a linear differential Hebbian learning system a method by which we can analytically investigate the dynamics and stability of the connections in recurrent networks. We use noisy periodic external input signals, which through the recurrent connections lead to complex actual ongoing inputs and observe that large stable ranges emerge in these networks without boundaries or weight-normalization. Somewhat counter-intuitively, we find that about 40% of these cases are obtained with a long-term potentiation-dominated STDP curve. Noise can reduce stability in some cases, but generally this does not occur. Instead stable domains are often enlarged. This study is a first step toward a better understanding of the ongoing interactions between activity and plasticity in recurrent networks using STDP. The results suggest that stability of (sub-)networks should generically be present also in larger structures.

Behavioral inhibition during reversal learning in the limbic system: a computational model

A biological model of the sub-cortical nuclei of the limbic system capable of performing reversal learning in a foodseeking task is presented. Contrary to other learning models, the reversal is modeled not by eliminating (unlearning) learned behaviors, but rather by the inhibiting the previously learned behavior. This allows for the re-instatement of behavior to be quickly re-established in such a way that animal behavior is simulated. In this model, the role of dopamine is different to standard models. The activity of dopaminergic (DA) neurons required for successful conditioning has commonly been identified as an error signal whereby an increased DA activity codes a positive error i.e. long term potentiation (LTP) and a decrease in dopamine concentration codes a negative error. In this model learning is achieved by implementing a form of differential Hebbian learning known as Isotropic Sequence Order learning and a third factor (ISO3). This third factor enables learning to be triggered at relevant moments. It is modeled by dopaminergic neurons which can be activated via a direct excitatory glutamatergic pathway, and an indirect disinhibitory GABAergic pathway. While the former generates phasic DA release which during acquisition enables long term potentiation (LTP) to occur, the latter produces in an increased population of tonically active DA neurons which generates long term depression (LTD) when an adjustment in learned behavior is required. The nucleus accumbens is divided into two shell and core subunits and has been modeled to function in distinct but complementary manners. Here, the core uses conditioned reinforcers to invigorate instrumental responding. The core has been modeled to learn to enable behavioral responding to reward predicting stimuli and as such undergoes minimal LTD so as not to eliminate these learned processes. On the other hand, LTD occurs in the shell that, through a shell-ventral pallido-medio dorsal pathway, influences the core and enables behavioral flexibility. A simple reversal-learning task will be used to demonstrate how this biophysically realistic pathway can be used to learn and reverse a simple food-seeking task. from Eighteenth Annual Computational Neuroscience Meeting: CNS*2009 Berlin, Germany. 18–23 July 2009

On the Asymptotic Equivalence Between Differential Hebbian and Temporal Difference Learning

In this theoretical contribution, we provide mathematical proof that two of the most important classes of network learning-correlation-based differential Hebbian learning and reward-based temporal difference learning-are asymptotically equivalent when timing the learning with a modulatory signal. This opens the opportunity to consistently reformulate most of the abstract reinforcement learning framework from a correlation-based perspective more closely related to the biophysics of neurons.

On the analysis of differential hebbian learning in closed-loop behavioral systems

On the analysis of differential hebbian learning in closed-loop behavioral systems

Mathematical properties of neuronal TD-rules and differential Hebbian learning: a comparison

A confusingly wide variety of temporally asymmetric learning rules exists related to reinforcement learning and/or to spike-timing dependent plasticity, many of which look exceedingly similar, while displaying strongly different behavior. These rules often find their use in control tasks, for example in robotics and for this rigorous convergence and numerical stability is required. The goal of this article is to review these rules and compare them to provide a better overview over their different properties. Two main classes will be discussed: temporal difference (TD) rules and correlation based (differential hebbian) rules and some transition cases. In general we will focus on neuronal implementations with changeable synaptic weights and a time-continuous representation of activity. In a machine learning (non-neuronal) context, for TD-learning a solid mathematical theory has existed since several years. This can partly be transfered to a neuronal framework, too. On the other hand, only now a more complete theory has also emerged for differential Hebb rules. In general rules differ by their convergence conditions and their numerical stability, which can lead to very undesirable behavior, when wanting to apply them. For TD, convergence can be enforced with a certain output condition assuring that the δ-error drops on average to zero (output control). Correlation based rules, on the other hand, converge when one input drops to zero (input control). Temporally asymmetric learning rules treat situations where incoming stimuli follow each other in time. Thus, it is necessary to remember the first stimulus to be able to relate it to the later occurring second one. To this end different types of so-called eligibility traces are being used by these two different types of rules. This aspect leads again to different properties of TD and differential Hebbian learning as discussed here. Thus, this paper, while also presenting several novel mathematical results, is mainly meant to provide a road map through the different neuronally emulated temporal asymmetrical learning rules and their behavior to provide some guidance for possible applications.

On the analysis and evaluation of closed loop learning systems

Event Abstract Back to Event On the analysis and evaluation of closed loop learning systems Tomas Kulvicius1*, Christoph Kolodziejski1, Paolo Di Prodi1, Minija Tamosiunaite1, Bernd Porr1 and Florentin Wörgötter1 1 BCCN Göttingen, Germany Understanding closed loop behavioral systems is a non-trivial problem, especially when they change with learning. Shannon's information theory is mostly used to analyze open loop systems. Only a few attempts exist which analyze systems from information point of view in a closed loop context, mostly measuring information of inputs. In this study we analyze closed loop systems by looking at the input as well as the output space. Specifically we investigate systems that perform differential Hebbian learning (STDP) and implement this learning rule in simulated agents. We observe that, fundamentally, there exist only two types of such closed loop systems: 1) Avoidance systems which increase time difference between far (predictive) and near (reflexive) sensory inputs, e.g. obstacle avoidance, and 2) Attraction systems for which the time difference is decreased, e.g. food retrieval. We show that analytical solutions can be found for the temporal development of such systems for simple cases. In the second part of this study we try to answer the following two questions: 1) How can we evaluate the success of learning, and 2) How can we find an optimal agent for a specific environment. These questions are addressed using energy and entropy measures and investigating their development during learning. This way we can show that within well-specified scenarios there are indeed agents which are optimal with respect to their structure and adaptive properties. Thus, this study is an attempt to address the difficult problem of quantifying non-stationary (due to the learning) closed-loop system, required for a better understanding of the complex dynamics of such systems. Conference: Bernstein Symposium 2008, Munich, Germany, 8 Oct - 10 Oct, 2008. Presentation Type: Poster Presentation Topic: All Abstracts Citation: Kulvicius T, Kolodziejski C, Prodi P, Tamosiunaite M, Porr B and Wörgötter F (2008). On the analysis and evaluation of closed loop learning systems. Front. Comput. Neurosci. Conference Abstract: Bernstein Symposium 2008. doi: 10.3389/conf.neuro.10.2008.01.078 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 17 Nov 2008; Published Online: 17 Nov 2008. * Correspondence: Tomas Kulvicius, BCCN Göttingen, Göttingen, Germany, tomas@nld.ds.mpg.de Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Tomas Kulvicius Christoph Kolodziejski Paolo Di Prodi Minija Tamosiunaite Bernd Porr Florentin Wörgötter Google Tomas Kulvicius Christoph Kolodziejski Paolo Di Prodi Minija Tamosiunaite Bernd Porr Florentin Wörgötter Google Scholar Tomas Kulvicius Christoph Kolodziejski Paolo Di Prodi Minija Tamosiunaite Bernd Porr Florentin Wörgötter PubMed Tomas Kulvicius Christoph Kolodziejski Paolo Di Prodi Minija Tamosiunaite Bernd Porr Florentin Wörgötter Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Self-influencing synaptic plasticity: Recurrent changes of synaptic weights can lead to specific functional properties

Recent experimental results suggest that dendritic and back-propagating spikes can influence synaptic plasticity in different ways (Holthoff, 2004; Holthoff et al., 2005). In this study we investigate how these signals could interact at dendrites in space and time leading to changing plasticity properties at local synapse clusters. Similar to a previous study (Saudargiene et al., 2004) we employ a differential Hebbian learning rule to emulate spike-timing dependent plasticity and investigate how the interaction of dendritic and back-propagating spikes, as the post-synaptic signals, could influence plasticity. Specifically, we will show that local synaptic plasticity driven by spatially confined dendritic spikes can lead to the emergence of synaptic clusters with different properties. If one of these clusters can drive the neuron into spiking, plasticity may change and the now arising global influence of a back-propagating spike can lead to a further segregation of the clusters and possibly the dying-off of some of them leading to more functional specificity. These results suggest that through plasticity being a spatial and temporal local process, the computational properties of dendrites or complete neurons can be substantially augmented.

A Methodology for Developing Adaptive Fuzzy Cognitive Maps for Decision Support

Differential Hebbian Learning (DHL) was proposed by Kosko as an unsupervised learning scheme for Fuzzy Cognitive Maps (FCMs). DHL can be used with a sequence of state vectors to adapt the causal link strengths of an FCM. However, it does not guarantee learning of the sequence by the FCM and no concrete procedures for the use of DHL has been developed. In this paper a formal methodology is proposed for using DHL in the development of FCMs in a decision support context. The four steps in the methodology are: (1) Creation of a crisp cognitive map; (2) Identification of event sequences for use in DHL; (3) Event sequence encoding using DHL; (4) Revision of the trained FCM. Feasibility of the proposed methodology is demonstrated with an example involving a dynamic system with feedback based on a real-life scenario.