Brain-Inspired Spiking Neural Network Research Articles

Brain-Inspired Spatio-Temporal Associative Memories for Neuroimaging Data Classification: EEG and fMRI.

Humans learn from a lot of information sources to make decisions. Once this information is learned in the brain, spatio-temporal associations are made, connecting all these sources (variables) in space and time represented as brain connectivity. In reality, to make a decision, we usually have only part of the information, either as a limited number of variables, limited time to make the decision, or both. The brain functions as a spatio-temporal associative memory. Inspired by the ability of the human brain, a brain-inspired spatio-temporal associative memory was proposed earlier that utilized the NeuCube brain-inspired spiking neural network framework. Here we applied the STAM framework to develop STAM for neuroimaging data, on the cases of EEG and fMRI, resulting in STAM-EEG and STAM-fMRI. This paper showed that once a NeuCube STAM classification model was trained on a complete spatio-temporal EEG or fMRI data, it could be recalled using only part of the time series, or/and only part of the used variables. We evaluated both temporal and spatial association and generalization accuracy accordingly. This was a pilot study that opens the field for the development of classification systems on other neuroimaging data, such as longitudinal MRI data, trained on complete data but recalled on partial data. Future research includes STAM that will work on data, collected across different settings, in different labs and clinics, that may vary in terms of the variables and time of data collection, along with other parameters. The proposed STAM will be further investigated for early diagnosis and prognosis of brain conditions and for diagnostic/prognostic marker discovery.

Brain-Inspired Spiking Neural Network Using Superconducting Devices

Based on recent research in artificial neural networks, researchers have focused on topics from brain-like computing based on the Von Neumann architecture to brain-inspired computing based on the integration of storage and calculation due to the large energy dissipation. Inspired by biological neural networks, people use superconducting devices, which have nonlinear dynamic characteristics, to construct synapse and neuron circuits. However, it is challenging to build large-scale superconducting spiking neural networks due to fan-ins and fan-outs of superconducting devices. In this paper, we present a criterion to scale up the superconducting neural network. Then, we present a three-layer fully connected superconducting spiking neural network construction scheme. Inspired by the biological neural network, we present a new training method based on frequency coding. After only 100 training iterations with one grayscale image of a digit, the learned three-layer fully connected superconducting spiking neural network has a recognition accuracy rate of 86.1% on the digit class. Moreover, the power dissipation of one spiking event in a superconducting synapse is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$23.2{\boldsymbol{zJ}}$</tex-math></inline-formula> , which shows that the network has outstandingly high efficiency over existing biologically inspired neural networks.

Spatiotemporal EEG Dynamics of Prospective Memory in Ageing and Mild Cognitive Impairment

Abstract Prospective memory (PM, the memory of future intentions) is one of the first complaints of those that develop dementia-related disease. Little is known about the neurophysiology of PM in ageing and those with mild cognitive impairment (MCI). By using a novel artificial neural network to investigate the spatial and temporal features of PM related brain activity, new insights can be uncovered. Young adults (n = 30), healthy older adults (n = 39) and older adults with MCI (n = 27) completed a working memory and two PM (perceptual, conceptual) tasks. Time-locked electroencephalographic potentials (ERPs) from 128-electrodes were analysed using a brain-inspired spiking neural network (SNN) architecture. Local and global connectivity from the SNNs was then evaluated. SNNs outperformed other machine learning methods in classification of brain activity between younger, older and older adults with MCI. SNNs trained using PM related brain activity had better classification accuracy than working memory related brain activity. In general, younger adults exhibited greater local cluster connectivity compared to both older adult groups. Older adults with MCI demonstrated decreased global connectivity in response to working memory and perceptual PM tasks but increased connectivity in the conceptual PM models relative to younger and healthy older adults. SNNs can provide a useful method for differentiating between those with and without MCI. Using brain activity related to PM in combination with SNNs may provide a sensitive biomarker for detecting cognitive decline. Cognitively demanding tasks may increase the amount connectivity in older adults with MCI as a means of compensation.

The spike gating flow: A hierarchical structure-based spiking neural network for online gesture recognition.

Action recognition is an exciting research avenue for artificial intelligence since it may be a game changer in emerging industrial fields such as robotic visions and automobiles. However, current deep learning (DL) faces major challenges for such applications because of the huge computational cost and inefficient learning. Hence, we developed a novel brain-inspired spiking neural network (SNN) based system titled spiking gating flow (SGF) for online action learning. The developed system consists of multiple SGF units which are assembled in a hierarchical manner. A single SGF unit contains three layers: a feature extraction layer, an event-driven layer, and a histogram-based training layer. To demonstrate the capability of the developed system, we employed a standard dynamic vision sensor (DVS) gesture classification as a benchmark. The results indicated that we can achieve 87.5% of accuracy which is comparable with DL, but at a smaller training/inference data number ratio of 1.5:1. Only a single training epoch is required during the learning process. Meanwhile, to the best of our knowledge, this is the highest accuracy among the non-backpropagation based SNNs. Finally, we conclude the few-shot learning (FSL) paradigm of the developed network: 1) a hierarchical structure-based network design involves prior human knowledge; 2) SNNs for content-based global dynamic feature detection.

SpikeBASE: Spiking Neural Learning Algorithm With Backward Adaptation of Synaptic Efflux

Brain-inspired Spiking Neural Network (SNN) is opening new possibilities towards human-level intelligence, by leveraging its nature of spatiotemporal information encoding and processing that bring both learning effectiveness and energy efficiency. Although substantial advances in SNN studies have been made, highly effective SNN learning algorithms are still urged, driven by the challenges of coordinating spiking spatiotemporal dynamics. We therefore propose a novel algorithm, SpikeBASE, denoting Spiking learning with Backward Adaption of Synaptic Efflux, to globally, supervisedly, and comprehensively coordinate the synaptic dynamics including both synaptic strength and responses. SpikeBASE can learn synaptic strength by backpropagating the error through the predefined synaptic responses. More importantly, SpikeBASE enables synaptic response adaptation through backpropagation, to mimic the complex dynamics of neural transmissions. Further, SpikeBASE enables multi-scale temporal memory formation by supporting multi-synaptic response adaptation. We have evaluated the algorithm on a challenging scarce data learning task and shown highly promising performance. The proposed SpikeBASE algorithm, through comprehensively coordinating the learning of synaptic strength, synaptic responses, and multi-scale temporal memory formation, has demonstrated its effectiveness on end-to-end SNN training. This study is expected to greatly advance the learning effectiveness of SNN and thus broadly benefit smart and efficient big data applications.

Spike-Based Approximate Backpropagation Algorithm of Brain-Inspired Deep SNN for Sonar Target Classification.

With the development of neuromorphic computing, more and more attention has been paid to a brain-inspired spiking neural network (SNN) because of its ultralow energy consumption and high-performance spatiotemporal information processing. Due to the discontinuity of the spiking neuronal activation function, it is still a difficult problem to train brain-inspired deep SNN directly, so SNN has not yet shown performance comparable to that of an artificial neural network. For this reason, the spike-based approximate backpropagation (SABP) algorithm and a general brain-inspired SNN framework are proposed in this paper. The combination of the two can be used for end-to-end direct training of brain-inspired deep SNN. Experiments show that compared with other spike-based methods of directly training SNN, the classification accuracy of this method is close to the best results on MNIST and CIFAR-10 datasets and achieves the best classification accuracy on sonar image target classification (SITC) of small sample datasets. Further analysis shows that compared with artificial neural networks, our brain-inspired SNN has great advantages in computational complexity and energy consumption in sonar target classification.

Hyperspectral Image Classification of Brain-Inspired Spiking Neural Network Based on Approximate Derivative Algorithm

Recently, deep learning methods have made significant progress in solving hyperspectral images (HSIs) classification problems of high-dimensional features, band redundancy, and spectral mixture. However, the deep neural network is too complex, with a long training time and high energy consumption, making it difficult to deploy on edge computing devices. In order to solve the above problems, this paper proposes a brain-inspired computing framework based on the spiking leaky integrate-and-fire neuron model for HSIs classification. Then we design an approximate derivative algorithm to solve the non-differentiable spike activity of the spiking neuron. The framework uses direct coding to generate spatiotemporal spikes for input HSI and achieves efficient extraction of spatial-spectral features through spiking standard convolution and spiking depthwise separable convolution. Extensive experiments are performed on four benchmark hyperspectral data sets and two public unmanned aerial vehicle-borne hyperspectral data sets. Experiments show that the proposed model has the advantages of high classification accuracy and fewer spiking time steps. The proposed model can save about 10 times computational energy consumption compared with the CNN of the same architecture. This research has great significance for overcoming the technical bottleneck of HSI classification based on brain-inspired computing, solving the critical problems of mobile computing in unmanned autonomous systems, and realizing the engineering application of unmanned aerial vehicles and software-defined satellites. The source code will be made available at https://github.com/Katherine-Cao/HSI_SNN.

Hyperspectral Image Classification of Brain-Inspired Spiking Neural Network Based on Attention Mechanism

Convolutional neural network has a complex model structure in hyperspectral image (HSI) classification and the energy consumption during training and inference is high, so it cannot be applied in edge computing devices such as software-defined satellites and unmanned aerial vehicles. In order to solve the classification of HSI in edge computing environment, inspired by the principle of neuro-dynamics and brain-inspired computing, we use integrate and fire neurons and shuffle squeeze and excitation network to construct a spiking neural network (SNN-SSEM). This letter designs an approximate derivative back propagation algorithm for discontinuous activation function and realizes the training of spiking neural network. Experiments were conducted on three HSI data sets and the average classification accuracy reached more than 99%. The energy consumption of our model is about 4.5 times that of convolutional neural network with the same architecture. This study is an exploration of the application of scientific theory of brain-inspired computing in hyperspectral remote sensing technology, which can realize real-time classification of HSI in mobile computing environment.

HuRAI: A brain-inspired computational model for human-robot auditory interface

The deep learning era endows immense opportunities for ubiquitous robotic applications by leveraging big data generated from widespread sensors and ever-growing computing capability. While the growing demands for natural human-robot interaction (HRI) as well as concerns for energy efficiency, real-time performance, and data security motive novel solutions. In this paper, we present a brain-inspired spiking neural network (SNN) based Human-Robot Auditory Interface, namely HuRAI. The HuRAI integrates the voice activity detection, speaker localization and voice command recognition systems into a unified framework that can be implemented on the emerging low-power neuromorphic computing (NC) devices. Our experimental results demonstrate superior modeling capabilities of SNNs, achieving accurate and rapid prediction for each task. Moreover, the energy efficiency analysis reveals a compelling prospect, with up to three orders of magnitude energy savings, over the equivalent artificial neural networks that running on the state-of-the-art Nvidia graphics processing unit (GPU). Therefore, integrating the algorithmic power of large-scale SNN models and the energy efficiency of NC devices offers an attractive solution for real-time, low-power robotic applications.

Quantum superposition inspired spiking neural network

SummaryDespite advances in artificial intelligence models, neural networks still cannot achieve human performance, partly due to differences in how information is encoded and processed compared with human brain. Information in an artificial neural network (ANN) is represented using a statistical method and processed as a fitting function, enabling handling of structural patterns in image, text, and speech processing. However, substantial changes to the statistical characteristics of the data, for example, reversing the background of an image, dramatically reduce the performance. Here, we propose a quantum superposition spiking neural network (QS-SNN) inspired by quantum mechanisms and phenomena in the brain, which can handle reversal of image background color. The QS-SNN incorporates quantum theory with brain-inspired spiking neural network models from a computational perspective, resulting in more robust performance compared with traditional ANN models, especially when processing noisy inputs. The results presented here will inform future efforts to develop brain-inspired artificial intelligence.

Deep Learning of Explainable EEG Patterns as Dynamic Spatiotemporal Clusters and Rules in a Brain-Inspired Spiking Neural Network

The paper proposes a new method for deep learning and knowledge discovery in a brain-inspired Spiking Neural Networks (SNN) architecture that enhances the model’s explainability while learning from streaming spatiotemporal brain data (STBD) in an incremental and on-line mode of operation. This led to the extraction of spatiotemporal rules from SNN models that explain why a certain decision (output prediction) was made by the model. During the learning process, the SNN created dynamic neural clusters, captured as polygons, which evolved in time and continuously changed their size and shape. The dynamic patterns of the clusters were quantitatively analyzed to identify the important STBD features that correspond to the most activated brain regions. We studied the trend of dynamically created clusters and their spike-driven events that occur together in specific space and time. The research contributes to: (1) enhanced interpretability of SNN learning behavior through dynamic neural clustering; (2) feature selection and enhanced accuracy of classification; (3) spatiotemporal rules to support model explainability; and (4) a better understanding of the dynamics in STBD in terms of feature interaction. The clustering method was applied to a case study of Electroencephalogram (EEG) data, recorded from a healthy control group (n = 21) and opiate use (n = 18) subjects while they were performing a cognitive task. The SNN models of EEG demonstrated different trends of dynamic clusters across the groups. This suggested to select a group of marker EEG features and resulted in an improved accuracy of EEG classification to 92%, when compared with all-feature classification. During learning of EEG data, the areas of neurons in the SNN model that form adjacent clusters (corresponding to neighboring EEG channels) were detected as fuzzy boundaries that explain overlapping activity of brain regions for each group of subjects.

Behavioral Outcomes and Neural Network Modeling of a Novel, Putative, Recategorization Sound Therapy.

The mechanisms underlying sound’s effect on tinnitus perception are unclear. Tinnitus activity appears to conflict with perceptual expectations of “real” sound, resulting in it being a salient signal. Attention diverted towards tinnitus during the later stages of object processing potentially disrupts high-order auditory streaming, and its uncertain nature results in negative psychological responses. This study investigated the benefits and neurophysiological basis of passive perceptual training and informational counseling to recategorize phantom perception as a more real auditory object. Specifically, it examined underlying psychoacoustic correlates of tinnitus and the neural activities associated with tinnitus auditory streaming and how malleable these are to change with targeted intervention. Eighteen participants (8 females, 10 males, mean age = 61.6 years) completed the study. The study consisted of 2 parts: (1) An acute exposure over 30 min to a sound that matched the person’s tinnitus (Tinnitus Avatar) that was cross-faded to a selected nature sound (Cicadas, Fan, Water Sound/Rain, Birds, Water and Bird). (2) A chronic exposure for 3 months to the same “morphed” sound. A brain-inspired spiking neural network (SNN) architecture was used to model and compare differences between electroencephalography (EEG) patterns recorded prior to morphing sound presentation, during, after (3-month), and post-follow-up. Results showed that the tinnitus avatar generated was a good match to an individual’s tinnitus as rated on likeness scales and was not rated as unpleasant. The five environmental sounds selected for this study were also rated as being appropriate matches to individuals’ tinnitus and largely pleasant to listen to. There was a significant reduction in the Tinnitus Functional Index score and subscales of intrusiveness of the tinnitus signal and ability to concentrate with the tinnitus trial end compared to baseline. There was a significant decrease in how strong the tinnitus signal was rated as well as ratings of how easy it was to ignore the tinnitus signal on severity rating scales. Qualitative analysis found that the environmental sound interacted with the tinnitus in a positive way, but participants did not experience change in severity, however, characteristics of tinnitus, including pitch and uniformity of sound, were reported to change. The results indicate the feasibility of the computational SNN method and preliminary evidence that the sound exposure may change activation of neural tinnitus networks and greater bilateral hemispheric involvement as the sound morphs over time into natural environmental sound; particularly relating to attention and discriminatory judgments (dorsal attention network, precentral gyrus, ventral anterior network). This is the first study that attempts to recategorize tinnitus using passive auditory training to a sound that morphs from resembling the person’s tinnitus to a natural sound. These findings will be used to design future-controlled trials to elucidate whether the approach used differs in effect and mechanism from conventional Broadband Noise (BBN) sound therapy.

Stylistic Composition of Melodies Based on a Brain-Inspired Spiking Neural Network.

Current neural network based algorithmic composition methods are very different compared to human brain's composition process, while the biological plausibility of composition and generative models are essential for the future of Artificial Intelligence. To explore this problem, this paper presents a spiking neural network based on the inspiration from brain structures and musical information processing mechanisms at multiple scales. Unlike previous methods, our model has three novel characteristics: (1) Inspired by brain structures, multiple brain regions with different cognitive functions, including musical memory and knowledge learning, are simulated and cooperated to generate stylistic melodies. A hierarchical neural network is constructed to formulate musical knowledge. (2) Biologically plausible neural model is employed to construct the network and synaptic connections are modulated using spike-timing-dependent plasticity (STDP) learning rule. Besides, brain oscillation activities with different frequencies perform importantly during the learning and generating process. (3) Based on significant musical memory and knowledge learning, genre-based and composer-based melody composition can be achieved by different neural circuits, the experiments show that the model can compose melodies with different styles of composers or genres.

Brain-inspired spiking neural networks for decoding and understanding muscle activity and kinematics from electroencephalography signals during hand movements

Compared to the abilities of the animal brain, many Artificial Intelligence systems have limitations which emphasise the need for a Brain-Inspired Artificial Intelligence paradigm. This paper proposes a novel Brain-Inspired Spiking Neural Network (BI-SNN) model for incremental learning of spike sequences. BI-SNN maps spiking activity from input channels into a high dimensional source-space which enhances the evolution of polychronising spiking neural populations. We applied the BI-SNN to predict muscle activity and kinematics from electroencephalography signals during upper limb functional movements. The BI-SNN extends our previously proposed eSPANNet computational model by integrating it with the ‘NeuCube’ brain-inspired SNN architecture. We show that BI-SNN can successfully predict continuous muscle activity and kinematics of upper-limb. The experimental results confirmed that the BI-SNN resulted in strongly correlated population activity and demonstrated the feasibility for real-time prediction. In contrast to the majority of Brain–Computer Interfaces (BCIs) that constitute a ‘black box’, BI-SNN provide quantitative and visual feedback about the related brain activity. This study is one of the first attempts to examine the feasibility of finding neural correlates of muscle activity and kinematics from electroencephalography using a brain-inspired computational paradigm. The findings suggest that BI-SNN is a better neural decoder for non-invasive BCI.

NeuroSense: Short-term emotion recognition and understanding based on spiking neural network modelling of spatio-temporal EEG patterns

Emotion recognition still poses a challenge lying at the core of the rapidly growing area of affective computing and is crucial for establishing a successful human–computer interaction. Identification and understanding of emotions are achieved through various measures, such as subjective self-reports, face-tracking, voice analysis, gaze-tracking, as well as the analysis of autonomic and central neurophysiological measurements. Current approaches to emotion recognition based on electroencephalography (EEG) mostly rely on various handcrafted features extracted over relatively long time windows of EEG during participants exposure to appropriate affective stimuli. In this paper, we present a short-term emotion recognition framework based on spiking neural network (SNN) modelling of spatio-temporal EEG patterns. Our method relies on EEG signal segmentation based on detection of short-term changes in facial landmarks, and as such includes no computation of handcrafted EEG features. Differences between participants’ EEG properties are taken into account via subject-dependent spike encoding in the formulated subject-independent emotion recognition task. We test our methods on the publicly available DEAP and MAHNOB-HCI databases due to the availability of both EEG and frontal face video data. Through an exhaustive hyperparameter optimisation strategy, we show that the proposed SNN-based representation of EEG spiking patterns provides valuable information for short- term emotion recognition. The obtained accuracies are 78.97% and 79.39% in arousal classification, and 67.76% and 72.12% in valence classification, on the DEAP and MAHNOB-HCI datasets, respectively. Furthermore, through the application of a brain-inspired SNN model, this study provides novel insight and helps in the understanding of the neural mechanisms involved in emotional processing in the context of audiovisual stimuli, such as affective videos. The presented results encourage the use of the proposed EEG processing methodology as a complement to existing features and methods commonly used for EEG-based emotion recognition, especially for short-term arousal recognition.

Prediction of Acoustic Residual Inhibition of Tinnitus Using a Brain-Inspired Spiking Neural Network Model.

Auditory Residual Inhibition (ARI) is a temporary suppression of tinnitus that occurs in some people following the presentation of masking sounds. Differences in neural response to ARI stimuli may enable classification of tinnitus and a tailored approach to intervention in the future. In an exploratory study, we investigated the use of a brain-inspired artificial neural network to examine the effects of ARI on electroencephalographic function, as well as the predictive ability of the model. Ten tinnitus patients underwent two auditory stimulation conditions (constant and amplitude modulated broadband noise) at two time points and were then characterised as responders or non-responders, based on whether they experienced ARI or not. Using a spiking neural network model, we evaluated concurrent neural patterns generated across space and time from features of electroencephalographic data, capturing the neural dynamic changes before and after stimulation. Results indicated that the model may be used to predict the effect of auditory stimulation on tinnitus on an individual basis. This approach may aid in the development of predictive models for treatment selection.

Bio-Inspired Self-Organizing Collision Avoidance for Drones Swarm Based on Reward-Modulated Spiking Neural Network

Biological systems can emerge intelligent swarm behavior through relatively simple individual, local interaction, and decentralized self-organizing decisionmaking. A major research challenge of self-organized swarm intelligence is the coupling influences between individual behaviors. Numerous swarm decisionmaking methods optimize the behavior of multiple individuals at the same time from a global perspective. Therefore, these methods are difficult to be used for online decision-making tasks of the real multi-robots system. This paper proposes a self-organized collision avoidance model for real drones incorporating a bio-inspired reward-modulated spiking neural network. Each individual independently uses the brain-inspired spiking neural network to update its own strategy according to the behavior of other agents. Subsequently, the local interaction and autonomous learning of a single individual lead to the emergence of swarm intelligence. We validated our model on swarm collision avoidance tasks in a bounded space, carrying out simulation and real-world experiments with a self-organized swarm of drones. Experimental results demonstrated that the swarm of UAVs without central control can exhibit collision avoidance even around perturbing obstacles.

Interpretability of Spatiotemporal Dynamics of the Brain Processes Followed by Mindfulness Intervention in a Brain-Inspired Spiking Neural Network Architecture.

Mindfulness training is associated with improvements in psychological wellbeing and cognition, yet the specific underlying neurophysiological mechanisms underpinning these changes are uncertain. This study uses a novel brain-inspired artificial neural network to investigate the effect of mindfulness training on electroencephalographic function. Participants completed a 4-tone auditory oddball task (that included targets and physically similar distractors) at three assessment time points. In Group A (n = 10), these tasks were given immediately prior to 6-week mindfulness training, immediately after training and at a 3-week follow-up; in Group B (n = 10), these were during an intervention waitlist period (3 weeks prior to training), pre-mindfulness training and post-mindfulness training. Using a spiking neural network (SNN) model, we evaluated concurrent neural patterns generated across space and time from features of electroencephalographic data capturing the neural dynamics associated with the event-related potential (ERP). This technique capitalises on the temporal dynamics of the shifts in polarity throughout the ERP and spatially across electrodes. Findings support anteriorisation of connection weights in response to distractors relative to target stimuli. Right frontal connection weights to distractors were associated with trait mindfulness (positively) and depression (inversely). Moreover, mindfulness training was associated with an increase in connection weights to targets (bilateral frontal, left frontocentral, and temporal regions only) and distractors. SNN models were superior to other machine learning methods in the classification of brain states as a function of mindfulness training. Findings suggest SNN models can provide useful information that differentiates brain states based on distinct task demands and stimuli, as well as changes in brain states as a function of psychological intervention.

Prediction of tinnitus masking benefit within a case series using a spiking neural network model.

Masking has been widely used as a tinnitus therapy, with large individual differences in its effectiveness. The basis of this variation is unknown. We examined individual tinnitus and psychological responses to three masking types, energetic masking (bilateral broadband static or rain noise [BBN]), informational masking (BBN with a notch at tinnitus pitch and 3-dimensional cues) and a masker combining both effects (BBN with spatial cues). Eleven participants with chronic tinnitus were followed for 12 months, each person used each masking approach for 3 months with a 1 month washout-baseline. The Tinnitus Functional Index (TFI), Tinnitus Rating Scales, Positive and Negative Affect Scale and Depression Anxiety Stress Scales, were measured every month of treatment. Electroencephalography (EEG) and psychoacoustic assessment was undertaken at baseline and following 3 months of each masking sound. The computational modeling of EEG data was based on the framework of brain-inspired Spiking Neural Network (SNN) architecture called NeuCube, designed for this study for mapping, learning, visualizing and classifying of brain activity patterns. EEG was related to clinically significant change in the TFI using the SNN model. The SNN framework was able to predict sound therapy responders (93% accuracy) from non-responders (100% accuracy) using baseline EEG recordings. The combination of energetic and informational masking was an effective treatment sound in more individuals than the other sounds used. Although the findings are promising, they are preliminary and require confirmation in independent and larger samples.

FusionSense: Emotion Classification Using Feature Fusion of Multimodal Data and Deep Learning in a Brain-Inspired Spiking Neural Network.

Using multimodal signals to solve the problem of emotion recognition is one of the emerging trends in affective computing. Several studies have utilized state of the art deep learning methods and combined physiological signals, such as the electrocardiogram (EEG), electroencephalogram (ECG), skin temperature, along with facial expressions, voice, posture to name a few, in order to classify emotions. Spiking neural networks (SNNs) represent the third generation of neural networks and employ biologically plausible models of neurons. SNNs have been shown to handle Spatio-temporal data, which is essentially the nature of the data encountered in emotion recognition problem, in an efficient manner. In this work, for the first time, we propose the application of SNNs in order to solve the emotion recognition problem with the multimodal dataset. Specifically, we use the NeuCube framework, which employs an evolving SNN architecture to classify emotional valence and evaluate the performance of our approach on the MAHNOB-HCI dataset. The multimodal data used in our work consists of facial expressions along with physiological signals such as ECG, skin temperature, skin conductance, respiration signal, mouth length, and pupil size. We perform classification under the Leave-One-Subject-Out (LOSO) cross-validation mode. Our results show that the proposed approach achieves an accuracy of 73.15% for classifying binary valence when applying feature-level fusion, which is comparable to other deep learning methods. We achieve this accuracy even without using EEG, which other deep learning methods have relied on to achieve this level of accuracy. In conclusion, we have demonstrated that the SNN can be successfully used for solving the emotion recognition problem with multimodal data and also provide directions for future research utilizing SNN for Affective computing. In addition to the good accuracy, the SNN recognition system is requires incrementally trainable on new data in an adaptive way. It only one pass training, which makes it suitable for practical and on-line applications. These features are not manifested in other methods for this problem.