Line-based event preprocessing: towards low-energy neuromorphic computer vision

Large-scale spiking neural networks (SNNs) have been used to successfully model complex neural circuits that explore various neural phenomena such as learning and memory, vision systems, auditory systems, neural oscillations, and many other important topics of neural function. Additionally, SNNs are particularly well-adapted to run on neuromorphic hardware as spiking events are often sparse, leading to a potentially large reduction in both bandwidth requirements and power usage. The inclusion of realistic plasticity equations, neural dynamics, and recurrent topologies has increased the descriptive power of SNNs but has also made the task of tuning these biologically realistic SNNs difficult. We present an automated parameter-tuning framework capable of tuning large-scale SNNs quickly and efficiently using evolutionary algorithms (EA) and off-the-shelf graphics processing units (GPUs).To test the feasibility of an automated parameter-tuning framework, our group used EAs to tune open parameters in SNNs running concurrently on a GPU. The SNNs were evolved to produce orientation-dependent stimulus responses similar to those found in simple cells of the primary visual cortex (V1) through the formation of self-organizing receptive fields (SORFs). The general evolutionary approach was as follows: A population of neural networks was created, each with a unique set of neural parameter values that defined overall behavior. Each SNN was then ranked based on a fitness value assigned by an objective function in which higher fitness values were given to SNNs that (a) reproduced responses observed in primate visual cortex, and (b) spanned the stimulus space, and (c) had sparse firing rates. The highest ranked individuals were selected, recombined, and mutated to form the offspring for the next generation. This process continued until a desired fitness was reached or until other termination conditions were met (Figure 1a).The automated parameter-tuning framework consisted of three software packages. The framework included: (a) the CARLsim SNN simulator [1], (2) the Evolving Objects (EO) computational framework [2], and (3) a parameter-tuning interface (PTI), developed by our group, to provide an interface between CARLsim and EO (See Figure 1b). The EO computational framework ran the evolutionary algorithm on the user-designated parameters of SNNs in CARLsim. The PTI allowed the objective function to be calculated independent of the EO computation framework. Parameter values were passed from the EO computation framework through the PTI to the SNN in CARLsim where the objective function is calculated. After the objective function was executed, the results were passed from the SNN in CARLsim through the PTI back to the EO computation framework for processing by the EA. With this approach, the fitness function calculation, which involved running each SNN in the population, could be run in parallel on the GPU while the remainder of EA calculations can be performed using the CPU (Figure 1b).A sample SNN with 4,104 neurons was tuned to respond with V1 simple cell-like tuning curves and produce SORFs. A performance analysis comparing the GPU-accelerated implementation to a single-threaded CPU implementation was carried out and showed that the GPU implementation could achieve a 65 times speedup over the CPU implementation. Additionally, the parameter value solutions found in the tuned SNN were stable and robust.The automated parameter-tuning framework presented here will be of use to both the computational neuroscience and neuromorphic engineering communities, making the process of constructing and tuning large-scale SNNs much quicker and easier.

Closed-loop brain-computer interfaces (BCIs) hold promise for restoring function after neurological damage by dynamically processing neural signals and delivering targeted brain stimulation. To achieve clinically meaningful outcomes, such systems must operate with high spatiotemporal precision. This work aims to demonstrate a proof-of-concept neuromorphic BCI that processes neural spike events in near-real time, without necessitating preprocessing besides signal filtering and spike detection. Methods - We developed a system that acquires neural signals and streams spike events into a spiking neural network (SNN) running on SpiNNaker neuromorphic hardware. We evaluated the system's performance using both in vivo recordings from mouse visual cortex and simulated neural waveforms. We measured the roundtrip latency, defined as the time from spike detection to an output spike generated by the SNN. Results - Under baseline conditions with no hidden SNN layers, mean roundtrip latency was 4.69 ms (±1.70 ms). Adding hidden layers increased latency by approximately 3.65 ms per layer, reflecting the computational overhead of deeper networks. The system successfully detected and processed spikes in near real-time, demonstrating that neuromorphic hardware can manage spike-based input at speeds suitable for closed-loop intervention. Discussion - These findings indicate that neuromorphic SNNs can rapidly process neural signals, providing a foundation for closed-loop BCIs capable of bypassing damaged neural pathways. Future efforts will involve implementing stimulation protocols and functional SNNs. Such developments may ultimately facilitate more effective, flexible, and power-efficient neuroprosthetic devices.

Recurrent spiking neural networks (SNNs) are inspired by the working principles of biological nervous systems that offer unique temporal dynamics and event-based processing. Recently, the error backpropagation through time (BPTT) algorithm has been successfully employed to train SNNs offline, with comparable performance to artificial neural networks (ANNs) on complex tasks. However, BPTT has severe limitations for online learning scenarios of SNNs where the network is required to simultaneously process and learn from incoming data. Specifically, as BPTT separates the inference and update phases, it would require to store all neuronal states for calculating the weight updates backwards in time. To address these fundamental issues, alternative credit assignment schemes are required. Within this context, neuromorphic hardware (NMHW) implementations of SNNs can greatly benefit from in-memory computing (IMC) concepts that follow the brain-inspired collocation of memory and processing, further enhancing their energy efficiency. In this work, we utilize a biologically-inspired local and online training algorithm compatible with IMC, which approximates BPTT, e-prop, and present an approach to support both inference and training of a recurrent SNN using NMHW. To do so, we embed the SNN weights on an in-memory computing NMHW with phase-change memory (PCM) devices and integrate it into a hardware-in-the-loop training setup. We develop our approach with respect to limited precision and imperfections of the analog devices using a PCM-based simulation framework and a NMHW consisting of in-memory computing cores fabricated in 14nm CMOS technology with 256×256 PCM crossbar arrays. We demonstrate that our approach is robust even to 4-bit precision and achieves competitive performance to a floating-point 32-bit realization, while simultaneously equipping the SNN with online training capabilities and exploiting the acceleration benefits of NMHW.

Neuromorphic hardware implements biological neurons and synapses to execute a spiking neural network (SNN)-based machine learning. We present SpiNeMap, a design methodology to map SNNs to crossbar-based neuromorphic hardware, minimizing spike latency and energy consumption. SpiNeMap operates in two steps: SpiNeCluster and SpiNePlacer. SpiNeCluster is a heuristic-based clustering technique to partition an SNN into clusters of synapses, where intracluster local synapses are mapped within crossbars of the hardware and intercluster global synapses are mapped to the shared interconnect. SpiNeCluster minimizes the number of spikes on global synapses, which reduces spike congestion and improves application performance. SpiNePlacer then finds the best placement of local and global synapses on the hardware using a metaheuristic-based approach to minimize energy consumption and spike latency. We evaluate SpiNeMap using synthetic and realistic SNNs on a state-of-the-art neuromorphic hardware. We show that SpiNeMap reduces average energy consumption by 45% and spike latency by 21%, compared to the best-performing SNN mapping technique.

Spiking Neural Networks (SNNs) are efficient computation models to perform spatio-temporal pattern recognition on resource- and power-constrained platforms. SNNs executed on neuromorphic hardware can further reduce energy consumption of these platforms. With increasing model size and complexity, mapping SNN-based applications to tile-based neuromorphic hardware is becoming increasingly challenging. This is attributed to the limitations of neuro-synaptic cores, viz. a crossbar, to accommodate only a fixed number of pre-synaptic connections per post-synaptic neuron. For complex SNN-based models that have many neurons and pre-synaptic connections per neuron, (1) connections may need to be pruned after training to fit onto the crossbar resources, leading to a loss in model quality, e.g., accuracy, and (2) the neurons and synapses need to be partitioned and placed on the neuro-sypatic cores of the hardware, which could lead to increased latency and energy consumption. In this work, we propose (1) a novel unrolling technique that decomposes a neuron function with many pre-synaptic connections into a sequence of homogeneous neural units to significantly improve the crossbar utilization and retain all pre-synaptic connections, and (2) SpiNeMap, a novel methodology to map SNNs on neuromorphic hardware with an aim to minimize energy consumption and spike latency.

Neuromorphic hardware, the new generation of non-von Neumann computing system, implements spiking neurons and synapses to spiking neural network (SNN)-based applications. The energy-efficient property makes the neuromorphic hardware suitable for power-constrained environments where sensors and edge nodes of the internet of things (IoT) work. The mapping of SNNs onto neuromorphic hardware is challenging because a non-optimized mapping may result in a high network-on-chip (NoC) latency and energy consumption. In this paper, we propose NeuMap, a simple and fast toolchain, to map SNNs onto the multicore neuromorphic hardware. NeuMap first obtains the communication patterns of an SNN by calculation that simplifies the mapping process. Then, NeuMap exploits localized connections, divides the adjacent layers into a sub-network, and partitions each sub-network into multiple clusters while meeting the hardware resource constraints. Finally, we employ a meta-heuristics algorithm to search for the best cluster-to-core mapping scheme in the reduced searching space. We conduct experiments using six realistic SNN-based applications to evaluate NeuMap and two prior works (SpiNeMap and SNEAP). The experimental results show that, compared to SpiNeMap and SNEAP, NeuMap reduces the average energy consumption by 84% and 17% and has 55% and 12% lower spike latency, respectively.

ABSTRACTIn the present study, the nonlinear material law of an in‐house FEM solver is substituted by physics‐informed neural networks (PINNs) and by physics‐based spiking neural networks (SNNs) for predicting the dynamic response of beams under impact loading. In the first part of this study concerning PINNs, the physics are included in the loss function of a deep neural network, also called a second‐generation neural network. The main advantage of using PINNs compared to classical ANNs is that they are considered data‐efficient function approximators. In the second part of this study, leaky integrate and fire (LIF) layers are deployed instead of dense layers in the PINN architecture, leading to a physics‐based third‐generation SNN, which enhances the PINN architecture in terms of energy efficiency. Third‐generation SNNs are inspired by the human brain function, where the information is processed via spikes. The spikes introduce sparse behavior, leading to a more energy‐efficient and sustainable deep learning method when implemented in neuromorphic hardware. The PINNs and physics‐based SNNs are employed in the implicit integration scheme of the material law of the in‐house FEM solver at the Gaussian level, and results are compared with the in‐house FEM solution in two different initial boundary value (IBV) problems. This study aims to show that PINNs can predict nonlinear material behavior and that the use of a physics‐based loss function enables the utility of less data in the training of the neural networks. Moreover, we show that we can enhance PINNs regarding energy consumption by adding a LIF layer. Therefore, the energy consumption is compared when the physics‐based SNN is deployed on the Loihi neuromorphic chip and on the CPU.

Spiking neural networks (SNN) are efficient computation models to infer spacio-temporal pattern recognition applications on neuromorphic hardware. Neuromorphic hardware are typically designed using interconnected crossbars, with each crossbar containing a structure of fully connected neurons. In order to ensure application performance such as accuracy and system performance such as throughput and resource utilization, SNNs need to be efficiently mapped on neuromorphic hardware. To address this, we propose a design flow to partition and map SNN-based applications on neuromorphic hardware, with an aim to enhance application and system performance. The design flow operates in two steps : (1) a two-step clustering technique to partition trained SNNs into clusters of neurons and synapses, with an aim to minimize inter-cluster spike communication, (2) mapping and scheduling the clusters on to crossbars-based architectures, modeled using Synchronous Data-flow Graphs (SDFGs). The SDFG model incorporates hardware constraints such as I/O bandwidth of crossbars and synaptic memory while analyzing the throughput of the modeled system. Our design-flow integrates CARLsim, a GPU-accelerated application-level SNN simulator with SDF3, a tool to map SDFG on hardware. We evaluate the design-flow using synthetic and realistic SNN-based applications. We show that, for throughput constrained applications, we achieve a 21.74% and 15.03% reduction in memory usage and utilization of the time-multiplexed interconnect, compared to a state of the art approach.

Large-scale spiking neural networks (SNNs) have been used to successfully model complex neural circuits that explore various neural phenomena such as learning and memory, vision systems, auditory systems, neural oscillations, and many other important topics of neural function. Additionally, SNNs are particularly well-adapted to run on neuromorphic hardware as spiking events are often sparse, leading to a potentially large reduction in both bandwidth requirements and power usage. The inclusion of realistic plasticity equations, neural dynamics, and recurrent topologies has increased the descriptive power of SNNs but has also made the task of tuning these biologically realistic SNNs difficult. We present an automated parameter-tuning framework capable of tuning large-scale SNNs quickly and efficiently using evolutionary algorithms (EA) and off-the-shelf graphics processing units (GPUs).

The field of neuromorphic engineering addresses the high energy demands of neural networks through brain-inspired hardware for efficient neural network computing. For on-chip learning with spiking neural networks, neuromorphic hardware requires a local learning algorithm able to solve complex tasks. Approaches based on burst-dependent plasticity have been proposed to address this requirement, but their ability to learn complex tasks has remained unproven. Specifically, previous burst-dependent learning was demonstrated on a spiking version of the ‘exclusive or’ problem (XOR) using a network of thousands of neurons. Here, we extend burst-dependent learning, termed ‘Burstprop’, to address more complex tasks with hundreds of neurons. We evaluate Burstprop on a rate-encoded spiking version of the MNIST dataset, achieving low test classification errors, comparable to those obtained using backpropagation through time on the same architecture. Going further, we develop another burst-dependent algorithm based on the communication of two types of error-encoding events for the communication of positive and negative errors. We find that this new algorithm performs better on the image classification benchmark. We also tested our algorithms under various types of feedback connectivity, establishing that the capabilities of fixed random feedback connectivity is preserved in spiking neural networks. Lastly, we tested the robustness of the algorithm to weight discretization. Together, these results suggest that spiking Burstprop can scale to more complex learning tasks and is therefore likely to be considered for self-supervised algorithms while maintaining efficiency, potentially providing a viable method for learning with neuromorphic hardware.

Spiking Neural Networks (SNN) promise extremely low-power and low-latency inference on neuromorphic hardware. Recent studies demonstrate the competitive performance of SNNs compared with Artificial Neural Networks (ANN) in conventional classification tasks.In this work, we present an energy-efficient implementation of a Reinforcement Learning (RL) algorithm using SNNs to solve an obstacle avoidance task performed by an Unmanned Aerial Vehicle (UAV), taking a Dynamic Vision Sensor (DVS) as event-based input. We train the SNN directly, improving upon state-of-art implementations based on hybrid (not directly trained) SNNs. For this purpose, we devise an adaptation of the Spatio-Temporal Backpropagation algorithm (STBP) for RL. We then compare the SNN with a state-of-art Convolutional Neural Network (CNN) designed to solve the same task. To this aim, we train both networks by exploiting a photorealistic training pipeline based on AirSim. To achieve a realistic latency and throughput assessment for embedded deployment, we designed and trained three different embedded SNN versions to be executed on state-of-art neuromorphic hardware, targeting state-of-the-art.We compared SNN and CNN in terms of obstacle avoidance performance showing that the SNN algorithm achieves better results than the CNN with a factor of 6× less energy. We also characterize the different SNN hardware implementations in terms of energy and spiking activity.

Spiking Neural Networks (SNNs) offer a biologically inspired computational paradigm that emulates neuronal activity through discrete spike-based processing. Despite their advantages, training SNNs with traditional backpropagation (BP) remains challenging due to computational inefficiencies and a lack of biological plausibility. This study explores the Forward-Forward (FF) algorithm as an alternative learning framework for SNNs. Unlike backpropagation, which relies on forward and backward passes, the FF algorithm employs two forward passes, enabling layer-wise localized learning, enhanced computational efficiency, and improved compatibility with neuromorphic hardware. We introduce an FF-based SNN training framework and evaluate its performance across both non-spiking (MNIST, Fashion-MNIST, Kuzushiji-MNIST) and spiking (Neuro-MNIST, SHD) datasets. Experimental results demonstrate that our model surpasses existing FF-based SNNs on evaluated static datasets with a much lighter architecture while achieving accuracy comparable to state-of-the-art backpropagation-trained SNNs. On more complex spiking tasks such as SHD, our approach outperforms other SNN models and remains competitive with leading backpropagation-trained SNNs. These findings highlight the FF algorithm's potential to advance SNN training methodologies by addressing some key limitations of backpropagation.

Artificial intelligence (AI) is revolutionising neuroimaging by enabling automated analysis, predictive analytics, and the discovery of biomarkers for neurological disorders. However, traditional artificial neural networks (ANNs) face challenges in processing spatiotemporal neuroimaging data due to their limited temporal memory and high computational demands. Spiking neural networks (SNNs), inspired by the brain’s biological processes, offer a promising alternative. SNNs use discrete spikes for event-driven communication, making them energy-efficient and well suited for the real-time processing of dynamic brain data. Among SNN architectures, NeuCube stands out as a powerful framework for analysing spatiotemporal neuroimaging data. It employs a 3D brain-like structure to model neural activity, enabling personalised modelling, disease classification, and biomarker discovery. This paper explores the advantages of SNNs and NeuCube for multimodal neuroimaging analysis, including their ability to handle complex spatiotemporal patterns, adapt to evolving data, and provide interpretable insights. We discuss applications in disease diagnosis, brain–computer interfaces, and predictive modelling, as well as challenges such as training complexity, data encoding, and hardware limitations. Finally, we highlight future directions, including hybrid ANN-SNN models, neuromorphic hardware, and personalised medicine. Our contributions in this work are as follows: (i) we give a comprehensive review of an SNN applied to neuroimaging analysis; (ii) we present current software and hardware platforms, which have been studied in neuroscience; (iii) we provide a detailed comparison of performance and timing of SNN software simulators with a curated ADNI and other datasets; (iv) we provide a roadmap to select a hardware/software platform based on specific cases; and (v) finally, we highlight a project where NeuCube has been successfully used in neuroscience. The paper concludes with discussions of challenges and future perspectives.

With the help of special neuromorphic hardware, spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption. It provides a promising energy-efficient way for realistic control tasks by combining SNNs with deep reinforcement learning (DRL). In this article, we focus on the task where the agent needs to learn multidimensional deterministic policies to control, which is very common in real scenarios. Recently, the surrogate gradient method has been utilized for training multilayer SNNs, which allows SNNs to achieve comparable performance with the corresponding deep networks in this task. Most existing spike-based reinforcement learning (RL) methods take the firing rate as the output of SNNs, and convert it to represent continuous action space (i.e., the deterministic policy) through a fully connected (FC) layer. However, the decimal characteristic of the firing rate brings the floating-point matrix operations to the FC layer, making the whole SNN unable to deploy on the neuromorphic hardware directly. To develop a fully spiking actor network (SAN) without any floating-point matrix operations, we draw inspiration from the nonspiking interneurons found in insects and employ the membrane voltage of the nonspiking neurons to represent the action. Before the nonspiking neurons, multiple population neurons are introduced to decode different dimensions of actions. Since each population is used to decode a dimension of action, we argue that the neurons in each population should be connected in time domain and space domain. Hence, the intralayer connections are used in output populations to enhance the representation capacity. This mechanism exists extensively in animals and has been demonstrated effectively. Finally, we propose a fully SAN with intralayer connections (ILC-SAN). Extensive experimental results demonstrate that the proposed method outperforms the state-of-the-art performance on continuous control tasks from OpenAI gym. Moreover, we estimate the theoretical energy consumption when deploying ILC-SAN on neuromorphic chips to illustrate its high energy efficiency.

Radio noise coming from artificial technology on Earth and, increasingly, in space, is inevitable, interferes with radio observations, and is difficult to detect and correct.Detecting and mitigating Radio Frequency Interference (RFI) is key to maximizing the scientific output of radio telescopes.RFI detection is a difficult observation task that requires an agent to distinguish genuine transient observations from radio interference.Hand-crafted algorithms suffice for current instruments, but as telescopes grow in their sensitivity so too grows their requirement to filter RFI.Ideal systems would be able to learn to detect novel types of RFI and flag otherwise anomolous observations.While machine learning techniques have shown promise in this task [1]-[4], the inability to conduct online learning efficiently and the associated energy cost in re-training models hampers using these techniques in practice, driving investigation into more efficient methods [5], [6].Spiking neural networks (SNNs) borrow more heavily from biological inspiration than Artificial Neural Networks (ANNs) and are well suited to time-varying data processing.While SNNs are more dynamic than equivalently sized ANNs, they are more difficult to train and difficult to simulate.Nascent neuromorphic hardware [7] implement SNNs directly, enabling enormous energy savings over traditional machine learning techniques if leveraged correctly.This work reports on an initial investigation into encoding complex visibility data into SNNs and outline avenues to test the efficacy of SNNs against traditional machine learning methods.SNNs and neuromorphic computing promise vast energy savings and better integration of continual learning; we outline how RFI detection in radio astronomy is an excellent candidate to test these claims.