Signal Denoising with Recurrent Spiking Neural Networks and Active Tuning

The problem of designing recurrent continuous-time and spiking neural networks is NP-Hard. A common practice is to utilize stochastic searches, such as evolutionary algorithms, to automatically construct acceptable networks. The outcome of the stochastic search is related to its ability to navigate the search space of neural networks and discover those of high quality. In this paper we investigate the search space associated with designing the above recurrent neural networks in order to differentiate which network should be easier to automatically design via a stochastic search. Our investigation utilizes two popular dynamic systems problems; (1) the Henon map and (2) the inverted pendulum as a benchmark.

Spiking neural networks are biologically plausible and power-efficient on neuromorphic hardware, while recurrent neural networks have been proven to be efficient on time series data. However, how to use the recurrent property to improve the performance of spiking neural networks is still a problem. This paper proposes a recurrent spiking neural network for character recognition using trajectories. In the network, a new encoding method is designed, in which varying time ranges of input streams are used in different recurrent layers. This is able to improve the generalization ability of our model compared with general encoding methods. The experiments are conducted on four groups of the character data set from University of Edinburgh. The results show that our method can achieve a higher average recognition accuracy than existing methods.

The emergence of brain-inspired neuromorphic computing as a paradigm for edge AI is motivating the search for high-performance and efficient spiking neural networks to run on this hardware. However, compared to classical neural networks in deep learning, current spiking neural networks lack competitive performance in compelling areas. Here, for sequential and streaming tasks, we demonstrate how a novel type of adaptive spiking recurrent neural network (SRNN) is able to achieve state-of-the-art performance compared to other spiking neural networks and almost reach or exceed the performance of classical recurrent neural networks (RNNs) while exhibiting sparse activity. From this, we calculate a $>$100x energy improvement for our SRNNs over classical RNNs on the harder tasks. To achieve this, we model standard and adaptive multiple-timescale spiking neurons as self-recurrent neural units, and leverage surrogate gradients and auto-differentiation in the PyTorch Deep Learning framework to efficiently implement backpropagation-through-time, including learning of the important spiking neuron parameters to adapt our spiking neurons to the tasks.

Path integration refers to the process by which animals continuously update spatial position during movement by integrating locomotor stimuli such as speed and head direction, thereby enabling real-time localization in continuous space. This process relies on the cooperation of spatial neurons, including grid cells and place cells. In recent years, continuous attractor networks and recurrent neural networks have provided useful insights into the mechanisms of path integration; however, they often rely on fixed-weight connections or exhibit a lack of biological plausibility. To explore more biologically plausible mechanisms, we propose a path integrator based on the recurrent spiking neural network (RSNN). Employing recurrently connected leaky integrate and fire (LIF) neurons, the RSNN encodes spatial positions as spike sequences via membrane potential reset mechanisms, enabling robust long-term path integration. Analysis reveals the spontaneous emergence of spatial and locomotor units, with some units exhibiting spatial-locomotor conjunctive properties, indicating synergistic computations underlying path integration. Ablation experiments confirm that stripe and border units have a larger effect on performance than other unit types under our experimental conditions. Under sparse spiking conditions, the network naturally develops diverse biologically inspired representations. The RSNN's performance provides novel insights into neuronal synergistic mechanisms in biological navigation, offering a biologically grounded framework for path integration modeling and contributing to the development of brain-inspired navigation algorithms.

Learning the underlying dynamics of periodic and aperiodic systems evolving over space and time provides substantial challenges across scientific fields. Despite recent advances in machine learning (ML), our understanding of its limits to learning, identifying, and emulating real physical systems remains incomplete. With the increasing complexity of dynamical systems and higher power consumption, ML systems take inspiration from the complex learning mechanisms of the mammalian brain, for biologically plausible learning. As we delve into the designing of brain- inspired ML systems, we deal with the challenges of the learning procedures, network sizes, energy efficiency and training complexities involved with it. It's crucial to characterize their performance in learning and controlling unknown dynamical systems to address them. We address these issues by designing sparsity aware learning mechanisms in feedback driven recurrent neural networks (RNNs), incorporate biologically plausible neuron models, and integrate computational neuroscience to build data driven AI systems. We explore brain-inspired neuronal dynamics to build AI systems that can mimic and classify dynamical systems. We propose full-FORCE (FF) SNN, Reservoir-in-reservoir (R-i-R) architecture, and SNN based applications learning, We pro- pose sparsity aware learning in RNNs, full-FORCE based sparsity regularization in recurrent spiking neural networks (RSNN) and an RSNN based BLE application for dynamical systems learning. Our proposed learning methods and architectures exhibit strong performance, noise robustness and energy efficiency in simulated and real-world adaptive system emulation. We evaluate bio signals, motion trajectories, Lorenz systems, real-world dynamic variable datasets, and other dynamical systems in this dissertation.

With recent advances in learning algorithms, recurrent networks of spiking neurons are achieving performance that is competitive with vanilla recurrent neural networks. However, these algorithms are limited to small networks of simple spiking neurons and modest-length temporal sequences, as they impose high memory requirements, have difficulty training complex neuron models and are incompatible with online learning. Here, we show how the recently developed Forward-Propagation Through Time (FPTT) learning combined with novel liquid time-constant spiking neurons resolves these limitations. Applying FPTT to networks of such complex spiking neurons, we demonstrate online learning of exceedingly long sequences while outperforming current online methods and approaching or outperforming offline methods on temporal classification tasks. The efficiency and robustness of FPTT enable us to directly train a deep and performant spiking neural network for joint object localization and recognition, demonstrating the ability to train large-scale dynamic and complex spiking neural network architectures. Memory efficient online training of recurrent spiking neural networks without compromising accuracy is an open challenge in neuromorphic computing. Yin and colleagues demonstrate that training a recurrent neural network consisting of so-called liquid time-constant spiking neurons using an algorithm called Forward-Propagation Through Time allows for online learning and state-of-the-art performance at a reduced computational cost compared with existing approaches.

In neural circuits, recurrent connectivity plays a crucial role in network function and stability. However, existing recurrent spiking neural networks (RSNNs) are often constructed by random connections without optimization. While RSNNs can produce rich dynamics that are critical for memory formation and learning, systemic architectural optimization of RSNNs is still an open challenge. We aim to enable systematic design of large RSNNs via a new scalable RSNN architecture and automated architectural optimization. We compose RSNNs based on a layer architecture called Sparsely-Connected Recurrent Motif Layer (SC-ML) that consists of multiple small recurrent motifs wired together by sparse lateral connections. The small size of the motifs and sparse inter-motif connectivity leads to an RSNN architecture scalable to large network sizes. We further propose a method called Hybrid Risk-Mitigating Architectural Search (HRMAS) to systematically optimize the topology of the proposed recurrent motifs and SC-ML layer architecture. HRMAS is an alternating two-step optimization process by which we mitigate the risk of network instability and performance degradation caused by architectural change by introducing a novel biologically-inspired "self-repairing" mechanism through intrinsic plasticity. The intrinsic plasticity is introduced to the second step of each HRMAS iteration and acts as unsupervised fast self-adaptation to structural and synaptic weight modifications introduced by the first step during the RSNN architectural "evolution." We demonstrate that the proposed automatic architecture optimization leads to significant performance gains over existing manually designed RSNNs: we achieve 96.44% on TI46-Alpha, 94.66% on N-TIDIGITS, 90.28% on DVS-Gesture, and 98.72% on N-MNIST. To the best of the authors' knowledge, this is the first work to perform systematic architecture optimization on RSNNs.

Recurrent spiking neural networks (SNNs) are inspired by the working principles of biological nervous systems that offer unique temporal dynamics and event-based processing. Recently, the error backpropagation through time (BPTT) algorithm has been successfully employed to train SNNs offline, with comparable performance to artificial neural networks (ANNs) on complex tasks. However, BPTT has severe limitations for online learning scenarios of SNNs where the network is required to simultaneously process and learn from incoming data. Specifically, as BPTT separates the inference and update phases, it would require to store all neuronal states for calculating the weight updates backwards in time. To address these fundamental issues, alternative credit assignment schemes are required. Within this context, neuromorphic hardware (NMHW) implementations of SNNs can greatly benefit from in-memory computing (IMC) concepts that follow the brain-inspired collocation of memory and processing, further enhancing their energy efficiency. In this work, we utilize a biologically-inspired local and online training algorithm compatible with IMC, which approximates BPTT, e-prop, and present an approach to support both inference and training of a recurrent SNN using NMHW. To do so, we embed the SNN weights on an in-memory computing NMHW with phase-change memory (PCM) devices and integrate it into a hardware-in-the-loop training setup. We develop our approach with respect to limited precision and imperfections of the analog devices using a PCM-based simulation framework and a NMHW consisting of in-memory computing cores fabricated in 14nm CMOS technology with 256×256 PCM crossbar arrays. We demonstrate that our approach is robust even to 4-bit precision and achieves competitive performance to a floating-point 32-bit realization, while simultaneously equipping the SNN with online training capabilities and exploiting the acceleration benefits of NMHW.

This paper aims to develop an accurate estimation technique for computing state of charge (SOC) of a lithium-ion battery using recurrent neural network algorithm. Nonlinear autoregressive with exogenous input (NARX) model is a well-known subclass of the recurrent neural network which has proven to be very effective and computationally rich for controlling dynamic system and predicting time series. However, the accuracy of recurrent NARX neural network depends on the amount of input and output order as well as a number of neurons in a hidden layer. Therefore, this study presents an improved recurrent NARX neural network based SOC estimation with particle swarm optimization (PSO) algorithm for finding the best value of input delays, feedback delays and a number of neurons in a hidden layer. The proposed model uses three most significant factor such as current, voltage and temperature without considering battery model. The model robustness is checked at low temperature (0°C), medium temperature (25°C), and high temperature (45°C). The US06 drive cycle is selected for model training and testing. The effectiveness of the proposed approach is compared with the back-propagation neural network (BPNN) optimized by PSO based on the SOC error, root mean square error (RMSE) and mean absolute error (MAE) and average execution time (AET). The results prove that the proposed model has higher estimation speed and achieves higher accuracy in reducing RMSE and MAE by 53% and 50% than BPNN based PSO model at 25°C.

In recent years, the application of deep learning models at the edge has gained attention. Typically, artificial neural networks (ANNs) are trained on graphics processing units (GPUs) and optimized for efficient execution on edge devices. Training ANNs directly at the edge is the next step with many applications such as the adaptation of models to specific situations like changes in environmental settings or optimization for individuals, e.g., optimization for speakers for speech processing. Also, local training can preserve privacy. Over the last few years, many algorithms have been developed to reduce memory footprint and computation. A specific challenge to train recurrent neural networks (RNNs) for processing sequential data is the need for the Back Propagation Through Time (BPTT) algorithm to store the network state of all time steps. This limitation is resolved by the biologically-inspired E-prop approach for training Spiking Recurrent Neural Networks (SRNNs). We implement the E-prop algorithm on a prototype of the SpiNNaker 2 neuromorphic system. A parallelization strategy is developed to split and train networks on the ARM cores of SpiNNaker 2 to make efficient use of both memory and compute resources. We trained an SRNN from scratch on SpiNNaker 2 in real-time on the Google Speech Command dataset for keyword spotting. We achieved an accuracy of 91.12% while requiring only 680 KB of memory for training the network with 25 K weights. Compared to other spiking neural networks with equal or better accuracy, our work is significantly more memory-efficient. In addition, we performed a memory and time profiling of the E-prop algorithm. This is used on the one hand to discuss whether E-prop or BPTT is better suited for training a model at the edge and on the other hand to explore architecture modifications to SpiNNaker 2 to speed up online learning. Finally, energy estimations predict that the SRNN can be trained on SpiNNaker2 with 12 times less energy than using a NVIDIA V100 GPU.

The use of robotic arms in various fields of human endeavor has increased over the years, and with recent advancements in artificial intelligence enabled by deep learning, they are increasingly being employed in medical applications like assistive robots for paralyzed patients with neurological disorders, welfare robots for the elderly, and prosthesis for amputees. However, robot arms tailored towards such applications are resource-constrained. As a result, deep learning with conventional artificial neural network (ANN) which is often run on GPU with high computational complexity and high power consumption cannot be handled by them. Neuromorphic processors, on the other hand, leverage spiking neural network (SNN) which has been shown to be less computationally complex and consume less power, making them suitable for such applications. Also, most robot arms unlike living agents that combine different sensory data to accurately perform a complex task, use uni-modal data which affects their accuracy. Conversely, multi-modal sensory data has been demonstrated to reach high accuracy and can be employed to achieve high accuracy in such robot arms. This paper presents the study of a multi-modal neurorobotic prosthetic arm control system based on recurrent spiking neural network. The robot arm control system uses multi-modal sensory data from visual (camera) and electromyography sensors, together with spike-based data processing on our previously proposed R-NASH neuromorphic processor to achieve robust accurate control of a robot arm with low power. The evaluation result using both uni-modal and multi-modal input data show that the multi-modal input achieves a more robust performance at 87%, compared to the uni-modal.

The synaptic-tagging-and-capture (STC) hypothesis formulates that at each synapse the concurrence of a tag with protein synthesis yields the maintenance of changes induced by synaptic plasticity. This hypothesis provides a biological principle underlying the synaptic consolidation of memories that is not verified for recurrent neural circuits. We developed a theoretical model integrating the mechanisms underlying the STC hypothesis with calcium-based synaptic plasticity in a recurrent spiking neural network. In the model, calcium-based synaptic plasticity yields the formation of strongly interconnected cell assemblies encoding memories, followed by consolidation through the STC mechanisms. Furthermore, we show for the first time that STC mechanisms modify the storage of memories such that after several hours memory recall is significantly improved. We identify two contributing processes: a merely time-dependent passive improvement, and an active improvement during recall. The described characteristics can provide a new principle for storing information in biological and artificial neural circuits.

Speech enhancement improves communication in noisy environments, affecting areas such as automatic speech recognition (ASR), hearing aids, and telecommunications. With these domains typically being power-constrained and event-based, and often requiring low latency, neuromorphic algorithms–particularly spiking neural networks (SNNs)–hold significant potential. However, current effective SNN solutions require a long temporal window to calculate Short Time Fourier Transforms (STFTs) and thus impose substantial latency, typically around 32 ms, which is too long for applications such as hearing aids. Inspired by the Dual-Path Recurrent Neural Network (DPRNN) in deep neural networks (DNNs), we develop a two-phase time-domain streaming SNN fframework for speech enhancement, named Dual-Path Spiking Neural Network (DPSNN). DPSNNs achieve low latency by replacing the STFT and inverse STFT (iSTFT) in traditional frequency-domain models with a learned convolutional encoder and decoder. In the DPSNN, the first phase uses Spiking Convolutional Neural Networks (SCNNs) to capture temporal contextual information, while the second phase uses Spiking Recurrent Neural Networks (SRNNs) to focus on frequency-related features. In addition, threshold-based activation suppression, along with L 1 regularization loss, is applied to specific non-spiking layers in DPSNNs to further improve their energy efficiency. Evaluating on the Voice Cloning Toolkit (VCTK) Corpus and Intel N-DNS Challenge dataset, our approach demonstrates excellent performance in speech objective metrics, along with the very low latency (approximately 5 ms) required for applications like hearing aids.

Increasing the capacity of recurrent neural networks (RNN) usually involves augmenting the size of the hidden layer, with significant increase of computational cost. Recurrent neural tensor networks (RNTN) increase capacity using distinct hidden layer weights for each word, but with greater costs in memory usage. In this paper, we introduce restricted recurrent neural tensor networks (r-RNTN) which reserve distinct hidden layer weights for frequent vocabulary words while sharing a single set of weights for infrequent words. Perplexity evaluations show that for fixed hidden layer sizes, r-RNTNs improve language model performance over RNNs using only a small fraction of the parameters of unrestricted RNTNs. These results hold for r-RNTNs using Gated Recurrent Units and Long Short-Term Memory.

As an important class of spiking neural networks (SNNs), recurrent spiking neural networks (RSNNs) possess great computational power and have been widely used for processing sequential data like audio and text. However, most RSNNs suffer from two problems. First, due to the lack of architectural guidance, random recurrent connectivity is often adopted, which does not guarantee good performance. Second, training of RSNNs is in general challenging, bottlenecking achievable model accuracy. To address these problems, we propose a new type of RSNN, skip-connected self-recurrent SNNs (ScSr-SNNs). Recurrence in ScSr-SNNs is introduced by adding self-recurrent connections to spiking neurons. The SNNs with self-recurrent connections can realize recurrent behaviors similar to those of more complex RSNNs, while the error gradients can be more straightforwardly calculated due to the mostly feedforward nature of the network. The network dynamics is enriched by skip connections between nonadjacent layers. Moreover, we propose a new backpropagation (BP) method, backpropagated intrinsic plasticity (BIP), to boost the performance of ScSr-SNNs further by training intrinsic model parameters. Unlike standard intrinsic plasticity rules that adjust the neuron's intrinsic parameters according to neuronal activity, the proposed BIP method optimizes intrinsic parameters based on the backpropagated error gradient of a well-defined global loss function in addition to synaptic weight training. Based on challenging speech, neuromorphic speech, and neuromorphic image data sets, the proposed ScSr-SNNs can boost performance by up to 2.85% compared with other types of RSNNs trained by state-of-the-art BP methods.