Reward-modulated Spike-timing-dependent Plasticity Research Articles

Complementary inverter and reward-modulated spike timing dependent plasticity circuit based on organic transistors

Organic materials are the key to the next generation of electric memory elements because of their low cost, high flexibility and high scalability. Solution-processed n-channel and p-channel organic field effect transistors (FETs) by spinning coating technique was demonstrated. We use based-poly (vinylcarbazole) (PVK) p-type transistor and based-6-phenyl-C61 butyric acid methyl ester (PCBM) n-type transistor as the channel of bottom gate structure FET, respectively, to fabricate the complementary inverter. The maximum voltage gain of the electrically isolated FET complementary inverter is up to 22, the power consumption is less than 3 μW, and the transition voltage is −4.2 V. In addition, our devices are able to implement “AND”, “NAND”, “OR”, and “NOT” gate logic well. But more than that, p-type transistor based on PVK and n-type transistor based on PCBM were combined with the memristor, reward-modulated spike timing dependent plasticity was implemented by introduced reward/punishment signals to guide synaptic weight updates. It shows great potential in the electronic application system with low cost and high energy efficiency.

Incorporating structural plasticity into self-organization recurrent networks for sequence learning.

Spiking neural networks (SNNs), inspired by biological neural networks, have received a surge of interest due to its temporal encoding. Biological neural networks are driven by multiple plasticities, including spike timing-dependent plasticity (STDP), structural plasticity, and homeostatic plasticity, making network connection patterns and weights to change continuously during the lifecycle. However, it is unclear how these plasticities interact to shape neural networks and affect neural signal processing. Here, we propose a reward-modulated self-organization recurrent network with structural plasticity (RSRN-SP) to investigate this issue. Specifically, RSRN-SP uses spikes to encode information, and incorporate multiple plasticities including reward-modulated spike timing-dependent plasticity (R-STDP), homeostatic plasticity, and structural plasticity. On the one hand, combined with homeostatic plasticity, R-STDP is presented to guide the updating of synaptic weights. On the other hand, structural plasticity is utilized to simulate the growth and pruning of synaptic connections. Extensive experiments for sequential learning tasks are conducted to demonstrate the representational ability of the RSRN-SP, including counting task, motion prediction, and motion generation. Furthermore, the simulations also indicate that the characteristics arose from the RSRN-SP are consistent with biological observations.

A Bio-Plausible Approach to Realizing Heat-Evoked Nociceptive Withdrawal Reflex on the Upper Limb of a Humanoid Robot

In this letter, we present a method for realizing the heat-evoked nociceptive withdrawal reflex (NWR) in the upper limb of a humanoid robot so that it can avoid the potential damage caused by noxious heat. We use a spiking neural network whose structure, encoding scheme, and form of information transmission mimic the reflex arc in humans to improve bio-plausibility. The proper synaptic strengths between the sensory neurons and the interneurons in the first two layers are learned using the bio-plausible reward-modulated spike timing-dependent plasticity learning algorithm. By monitoring the spikes from the motor neuron in the third layer, a reflex matching the intensity of the stimulation can be evoked. Experimental evaluations show that noxious heat stimulation can be detected online and evoke the NWR. The experiments on a full-size humanoid robot show that the method enables robots to avoid potential damage robustly with proper NWR, depending on the site and intensity of the stimulation. We also verify that the method takes advantage of the intrinsic characteristics of its neuromorphic encoding scheme to reproduce essential features of the NWR, e.g., spatial summation effect and temporal summation effect in humans. The improved bio-plausibility and the capability to reproduce the human-like feature.

Bio-plausible digital implementation of a reward modulated STDP synapse

Reward-modulated Spike-Timing-Dependent Plasticity (R-STDP) is a learning method for Spiking Neural Network (SNN) that makes use of an external learning signal to modulate the synaptic plasticity produced by Spike-Timing-Dependent Plasticity (STDP). Combining the advantages of reinforcement learning and the biological plausibility of STDP, online learning on SNN in real-world scenarios can be applied. This paper presents a fully digital architecture, implemented on an Field-Programmable Gate Array (FPGA), including the R-STDP learning mechanism in a SNN. The hardware results obtained are comparable to the software simulations results using the Brian2 simulator. The maximum error is of 0.083 when a 14-bits fix-point precision is used in realtime. The presented architecture shows an accuracy of 95% when tested in an obstacle avoidance problem on mobile robotics with a minimum use of resources.

A Brain-Inspired Theory of Mind Spiking Neural Network for Reducing Safety Risks of Other Agents.

Artificial Intelligence (AI) systems are increasingly applied to complex tasks that involve interaction with multiple agents. Such interaction-based systems can lead to safety risks. Due to limited perception and prior knowledge, agents acting in the real world may unconsciously hold false beliefs and strategies about their environment, leading to safety risks in their future decisions. For humans, we can usually rely on the high-level theory of mind (ToM) capability to perceive the mental states of others, identify risk-inducing errors, and offer our timely help to keep others away from dangerous situations. Inspired by the biological information processing mechanism of ToM, we propose a brain-inspired theory of mind spiking neural network (ToM-SNN) model to enable agents to perceive such risk-inducing errors inside others' mental states and make decisions to help others when necessary. The ToM-SNN model incorporates the multiple brain areas coordination mechanisms and biologically realistic spiking neural networks (SNNs) trained with Reward-modulated Spike-Timing-Dependent Plasticity (R-STDP). To verify the effectiveness of the ToM-SNN model, we conducted various experiments in the gridworld environments with random agents' starting positions and random blocking walls. Experimental results demonstrate that the agent with the ToM-SNN model selects rescue behavior to help others avoid safety risks based on self-experience and prior knowledge. To the best of our knowledge, this study provides a new perspective to explore how agents help others avoid potential risks based on bio-inspired ToM mechanisms and may contribute more inspiration toward better research on safety risks.

Dopamine and serotonin interplay for valence-based spatial learning.

Dopamine (DA) and serotonin (5-HT) are important neuromodulators of synaptic plasticity that have been linked to learning from positive or negative outcomes or valence-based learning. In the hippocampus, both affect long-term plasticity but play different roles in encoding uncertainty or predicted reward. DA has been related to positive valence, from reward consumption or avoidance behavior, and 5-HT to aversive encoding. We propose DA produces overall LTP while 5-HT elicits LTD. Here, we compare two reward-modulated spike timing-dependent plasticity (R-STDP) rules to describe the action of these neuromodulators. We examined their role in cognitive performance and flexibility for computational models of the Morris water maze task and reversal learning. Our results show that the interplay of DA and 5-HT improves learning performance and can explain experimental evidence. This study reinforces the importance of neuromodulation in determining the direction of plasticity.

A Low-Cost FPGA Implementation of Spiking Extreme Learning Machine With On-Chip Reward-Modulated STDP Learning

For embedded, mobile and edge-computing intelligent applications, this brief proposes a low-cost real-time neuromorphic hardware system of spiking Extreme Learning Machine (ELM) equipped with on-chip triplet-based reward-modulated spike-timing-dependent plasticity (R-STDP) learning capability. Our design employs a time-step pipelined dual-core architecture consisting of parallel computing unit arrays to improve processing speed, as well as a trace-assisting learning mechanism and on-the-fly hidden layer weight re-generators to significantly reduce hardware resource costs. Our architecture is scalable to different spiking ELM sizes under different tradeoffs among processing speed, recognition accuracy and resource costs. Tests showed that the on-chip triplet R-STDP learning capability can help to achieve relatively high recognition accuracies on our hardware system. An FPGA prototype with low logic and memory resource consumption was implemented, achieving 93% and 78.5% recognition accuracies on the MNIST and Fashion-MNIST image datasets, respectively, at a speed of 30 frames per second (fps) for inference and 22.5 fps for on-chip learning.

A Reconfigurable Two‐WSe2‐Transistor Synaptic Cell for Reinforcement Learning

Reward-modulated spike-timing-dependent plasticity (R-STDP) is a brain-inspired reinforcement learning (RL) rule, exhibiting potential for decision-making tasks and artificial general intelligence. However, the hardware implementation of the reward-modulation process in R-STDP usually requires complicated Si complementary metal-oxide-semiconductor (CMOS) circuit design that causes high power consumption and large footprint. Here, a design with two synaptic transistors (2T) connected in a parallel structure is experimentally demonstrated. The 2T unit based on WSe2 ferroelectric transistors exhibits reconfigurable polarity behavior, where one channel can be tuned as n-type and the other as p-type due to nonvolatile ferroelectric polarization. In this way, opposite synaptic weight update behaviors with multilevel (>6 bit) conductance states, ultralow nonlinearity (0.56/-1.23), and large Gmax /Gmin ratio of 30 are realized. By applying positive/negative reward to (anti-)STDP component of 2T cell, R-STDP learning rules are realized for training the spiking neural network and demonstrated to solve the classical cart-pole problem, exhibiting a way for realizing low-power (32 pJ per forward process) and highly area-efficient (100 µm2 ) hardware chip for reinforcement learning.

An autonomous learning mobile robot using biological reward modulate STDP

Recent studies have shown that biologically inspired Spiking Neural Networks (SNNs) has potentials for the mobile robot controls. Based on SNNs, an autonomous learning paradigm for controlling mobile robots is proposed in this work, which can learn specific tasks autonomously. A reward modulated spike-timing-dependent plasticity (R-STDP) learning algorithm is designed to aid implementing the autonomous learning paradigm. It can train the SNN under different environmental states and conditions. The obstacle avoidance in the synthetic and real environments is used as a robotic task example to verify the effectiveness of the proposed paradigm. Results show that the mobile robot can learn autonomously under different environmental conditions and is able to avoid obstacles after learning processes complete.

Gene essentiality prediction based on chaos game representation and spiking neural networks

Abstract Chaos game representation (CGR) is a useful one-to-one visualization tool to represent nucleotide sequences, in which both local and global patterns of nucleotides can be graphically described. Deep learning networks have been proved to achieve outstanding performance on feature extraction and image recognition. In this paper, we use convolutional spiking neural networks (SNNs) with reward-modulated spike-timing-dependent plasticity (R-STDP) learning rule to learn from the frequency matrix chaos game representation (FCGR) images of essential and non-essential genes of 32 bacteria in the DEG database and make intra-organism and cross-organism essential gene predictions. For intra-organism predictions, our highest accuracy(ACC) score is 0.90 and the average ACC is 0.78, and for cross-organism predictions, our highest ACC is 0.79 and the average ACC is 0.68. Compared with the results of traditional machine learning classifiers training with FCGR images or numerical fractal features pre-calculated from CGR representations, our intra-organism prediction results are much better for all the bacteria or most bacteria, respectively, indicating that our spiking neural networks can make better essential gene prediction by extracting the gene features directly from the FCGR images of essential and nonessential genes. Compared with essential gene prediction methods using gene sequence features and topological features, our cross-organism prediction results can achieve performance close to or even better than such methods, while requiring much fewer input features.

ECG Classification Algorithm Based on STDP and R-STDP Neural Networks for Real-Time Monitoring on Ultra Low-Power Personal Wearable Devices.

This paper presents a novel ECG classification algorithm for inclusion as part of real-time cardiac monitoring systems in ultra low-power wearable devices. The proposed solution is based on spiking neural networks which are the third generation of neural networks. In specific, we employ spike-timing dependent plasticity (STDP), and reward-modulated STDP (R-STDP), in which the model weights are trained according to the timings of spike signals, and reward or punishment signals. Experiments show that the proposed solution is suitable for real-time operation, achieves comparable accuracy with respect to previous methods, and more importantly, its energy consumption in real-time classification of ECG signals is significantly smaller. In specific, energy consumption is 1.78 μJ per beat, which is 2 to 9 orders of magnitude smaller than previous neural network based ECG classification methods.

SpykeTorch: Efficient Simulation of Convolutional Spiking Neural Networks With at Most One Spike per Neuron.

Application of deep convolutional spiking neural networks (SNNs) to artificial intelligence (AI) tasks has recently gained a lot of interest since SNNs are hardware-friendly and energy-efficient. Unlike the non-spiking counterparts, most of the existing SNN simulation frameworks are not practically efficient enough for large-scale AI tasks. In this paper, we introduce SpykeTorch, an open-source high-speed simulation framework based on PyTorch. This framework simulates convolutional SNNs with at most one spike per neuron and the rank-order encoding scheme. In terms of learning rules, both spike-timing-dependent plasticity (STDP) and reward-modulated STDP (R-STDP) are implemented, but other rules could be implemented easily. Apart from the aforementioned properties, SpykeTorch is highly generic and capable of reproducing the results of various studies. Computations in the proposed framework are tensor-based and totally done by PyTorch functions, which in turn brings the ability of just-in-time optimization for running on CPUs, GPUs, or Multi-GPU platforms.

Indirect and direct training of spiking neural networks for end-to-end control of a lane-keeping vehicle

Building spiking neural networks (SNNs) based on biological synaptic plasticities holds a promising potential for accomplishing fast and energy-efficient computing, which is beneficial to mobile robotic applications. However, the implementations of SNNs in robotic fields are limited due to the lack of practical training methods. In this paper, we therefore introduce both indirect and direct end-to-end training methods of SNNs for a lane-keeping vehicle. First, we adopt a policy learned using the Deep Q-Learning (DQN) algorithm and then subsequently transfer it to an SNN using supervised learning. Second, we adopt the reward-modulated spike-timing-dependent plasticity (R-STDP) for training SNNs directly, since it combines the advantages of both reinforcement learning and the well-known spike-timing-dependent plasticity (STDP). We examine the proposed approaches in three scenarios in which a robot is controlled to keep within lane markings by using an event-based neuromorphic vision sensor. We further demonstrate the advantages of the R-STDP approach in terms of the lateral localization accuracy and training time steps by comparing them with other three algorithms presented in this paper.

Bio-inspired digit recognition using reward-modulated spike-timing-dependent plasticity in deep convolutional networks

The primate visual system has inspired the development of deep artificial neural networks, which have revolutionized the computer vision domain. Yet these networks are much less energy-efficient than their biological counterparts, and they are typically trained with backpropagation, which is extremely data-hungry. To address these limitations, we used a deep convolutional spiking neural network (DCSNN) and a latency-coding scheme. We trained it using a combination of spike-timing-dependent plasticity (STDP) for the lower layers and reward-modulated STDP (R-STDP) for the higher ones. In short, with R-STDP a correct (resp. incorrect) decision leads to STDP (resp. anti-STDP). This approach led to an accuracy of 97.2% on MNIST, without requiring an external classifier. In addition, we demonstrated that R-STDP extracts features that are diagnostic for the task at hand, and discards the other ones, whereas STDP extracts any feature that repeats. Finally, our approach is biologically plausible, hardware friendly, and energy-efficient.

Supervised Learning in SNN via Reward-Modulated Spike-Timing-Dependent Plasticity for a Target Reaching Vehicle.

Spiking neural networks (SNNs) offer many advantages over traditional artificial neural networks (ANNs) such as biological plausibility, fast information processing, and energy efficiency. Although SNNs have been used to solve a variety of control tasks using the Spike-Timing-Dependent Plasticity (STDP) learning rule, existing solutions usually involve hard-coded network architectures solving specific tasks rather than solving different kinds of tasks generally. This results in neglecting one of the biggest advantages of ANNs, i.e., being general-purpose and easy-to-use due to their simple network architecture, which usually consists of an input layer, one or multiple hidden layers and an output layer. This paper addresses the problem by introducing an end-to-end learning approach of spiking neural networks constructed with one hidden layer and reward-modulated Spike-Timing-Dependent Plasticity (R-STDP) synapses in an all-to-all fashion. We use the supervised reward-modulated Spike-Timing-Dependent-Plasticity learning rule to train two different SNN-based sub-controllers to replicate a desired obstacle avoiding and goal approaching behavior, provided by pre-generated datasets. Together they make up a target-reaching controller, which is used to control a simulated mobile robot to reach a target area while avoiding obstacles in its path. We demonstrate the performance and effectiveness of our trained SNNs to achieve target reaching tasks in different unknown scenarios.

First-Spike-Based Visual Categorization Using Reward-Modulated STDP.

Reinforcement learning (RL) has recently regained popularity with major achievements such as beating the European game of Go champion. Here, for the first time, we show that RL can be used efficiently to train a spiking neural network (SNN) to perform object recognition in natural images without using an external classifier. We used a feedforward convolutional SNN and a temporal coding scheme where the most strongly activated neurons fire first, while less activated ones fire later, or not at all. In the highest layers, each neuron was assigned to an object category, and it was assumed that the stimulus category was the category of the first neuron to fire. If this assumption was correct, the neuron was rewarded, i.e., spike-timing-dependent plasticity (STDP) was applied, which reinforced the neuron's selectivity. Otherwise, anti-STDP was applied, which encouraged the neuron to learn something else. As demonstrated on various image data sets (Caltech, ETH-80, and NORB), this reward-modulated STDP (R-STDP) approach has extracted particularly discriminative visual features, whereas classic unsupervised STDP extracts any feature that consistently repeats. As a result, R-STDP has outperformed STDP on these data sets. Furthermore, R-STDP is suitable for online learning and can adapt to drastic changes such as label permutations. Finally, it is worth mentioning that both feature extraction and classification were done with spikes, using at most one spike per neuron. Thus, the network is hardware friendly and energy efficient.

Sameness/difference spiking neural circuit as a relational concept precursor model: A bio-inspired robotic implementation

Abstract This paper presents a bio-inspired virtual and physical robotic model of sameness/difference (SD) abstract relations investigated through associative learning tasks. Considering that the invertebrate bees presumably possess this higher cognitive ability with a relatively tiny brain (Giurfa, Zhang, Jenett, Menzel, & Srinivasan, 2001), an hypothesis of a neural correlate may hold as a minimalist cellular circuit. In order to simulate it, we implemented the model using an artificial spiking neural network (SNN) acting as a robot’s brain-controller and a reward-modulated spike-timing dependent plasticity as the learning rule. The model is tested with different operant conditioning procedures of delayed-matching-to-sample and delayed-non-matching-to-sample tasks from dual choices of colors. Apart SD, the color and side simpler associative rules also co-exist. This allows the robot to learn in different scenarios, not knowing the rule to come. Therefore, we showed that depending on its action and the applied reinforcing rules (SD, color or side), the robot dynamically learns any of it, independently and one after the other, all supported by a single SNN. We believe that this SD SNN model could be used as a precursor base to understand and build a generic form of a relational concept mechanism.

Toward Building Hybrid Biological/in silico Neural Networks for Motor Neuroprosthetic Control

In this article, we introduce the Bioinspired Neuroprosthetic Design Environment (BNDE) as a practical platform for the development of novel brain–machine interface (BMI) controllers, which are based on spiking model neurons. We built the BNDE around a hard real-time system so that it is capable of creating simulated synapses from extracellularly recorded neurons to model neurons. In order to evaluate the practicality of the BNDE for neuroprosthetic control experiments, a novel, adaptive BMI controller was developed and tested using real-time closed-loop simulations. The present controller consists of two in silico medium spiny neurons, which receive simulated synaptic inputs from recorded motor cortical neurons. In the closed-loop simulations, the recordings from the cortical neurons were imitated using an external, hardware-based neural signal synthesizer. By implementing a reward-modulated spike timing-dependent plasticity rule, the controller achieved perfect target reach accuracy for a two-target reaching task in one-dimensional space. The BNDE combines the flexibility of software-based spiking neural network (SNN) simulations with powerful online data visualization tools and is a low-cost, PC-based, and all-in-one solution for developing neurally inspired BMI controllers. We believe that the BNDE is the first implementation, which is capable of creating hybrid biological/in silico neural networks for motor neuroprosthetic control and utilizes multiple CPU cores for computationally intensive real-time SNN simulations.

Emergence of task-dependent representations in working memory circuits.

A wealth of experimental evidence suggests that working memory circuits preferentially represent information that is behaviorally relevant. Still, we are missing a mechanistic account of how these representations come about. Here we provide a simple explanation for a range of experimental findings, in light of prefrontal circuits adapting to task constraints by reward-dependent learning. In particular, we model a neural network shaped by reward-modulated spike-timing dependent plasticity (r-STDP) and homeostatic plasticity (intrinsic excitability and synaptic scaling). We show that the experimentally-observed neural representations naturally emerge in an initially unstructured circuit as it learns to solve several working memory tasks. These results point to a critical, and previously unappreciated, role for reward-dependent learning in shaping prefrontal cortex activity.

A Spiking Network Model of Decision Making Employing Rewarded STDP

Reward-modulated spike timing dependent plasticity (STDP) combines unsupervised STDP with a reinforcement signal that modulates synaptic changes. It was proposed as a learning rule capable of solving the distal reward problem in reinforcement learning. Nonetheless, performance and limitations of this learning mechanism have yet to be tested for its ability to solve biological problems. In our work, rewarded STDP was implemented to model foraging behavior in a simulated environment. Over the course of training the network of spiking neurons developed the capability of producing highly successful decision-making. The network performance remained stable even after significant perturbations of synaptic structure. Rewarded STDP alone was insufficient to learn effective decision making due to the difficulty maintaining homeostatic equilibrium of synaptic weights and the development of local performance maxima. Our study predicts that successful learning requires stabilizing mechanisms that allow neurons to balance their input and output synapses as well as synaptic noise.