Neural Decoding Research Articles

Music-oriented auditory attention detection from electroencephalogram

Music-oriented auditory attention detection (AAD) aims at determining which instrument in polyphonic music a listener is paying attention to by analyzing the listener’s electroencephalogram (EEG). However, the existing linear models cannot effectively mimic the nonlinearity of the human brain, resulting in limited performance. Thus, a nonlinear music-oriented AAD model is proposed in this paper. Firstly, an auditory feature and a musical feature are fused to represent musical sources precisely and comprehensively. Secondly, the EEG is enhanced if music stimuli are presented in stereo. Thirdly, a neural network architecture is constructed to capture nonlinear and dynamic interactions between the EEG and auditory stimuli. Finally, the musical source most similar to the EEG in the common embedding space is identified as the attended one. Experimental results demonstrate that the proposed model outperforms all baseline models. On 1-s decision windows, it reaches accuracies of 92.6% and 81.7% under mono duo and trio stimuli, respectively. Additionally, it can be easily extended to speech-oriented AAD. This work can open up new possibilities for studies on both brain neural activity decoding and music information retrieval.

An empirical comparison of deep learning explainability approaches for EEG using simulated ground truth

Recent advancements in machine learning and deep learning (DL) based neural decoders have significantly improved decoding capabilities using scalp electroencephalography (EEG). However, the interpretability of DL models remains an under-explored area. In this study, we compared multiple model explanation methods to identify the most suitable method for EEG and understand when some of these approaches might fail. A simulation framework was developed to evaluate the robustness and sensitivity of twelve back-propagation-based visualization methods by comparing to ground truth features. Multiple methods tested here showed reliability issues after randomizing either model weights or labels: e.g., the saliency approach, which is the most used visualization technique in EEG, was not class or model-specific. We found that DeepLift was consistently accurate as well as robust to detect the three key attributes tested here (temporal, spatial, and spectral precision). Overall, this study provides a review of model explanation methods for DL-based neural decoders and recommendations to understand when some of these methods fail and what they can capture in EEG.

Mediodorsal thalamus-projecting anterior cingulate cortex neurons modulate helping behavior in mice

Many species living in groups can perform prosocial behaviors via voluntarily helping others with or without benefits for themselves. To provide a better understanding of the neural basis of such prosocial behaviors, we adapted a preference lever-switching task in which mice can prevent harm to others by switching from using a lever that causes shocks to a conspecific one that does not. We found the harm avoidance behavior was mediated by self-experience and visual and social contact but not by gender or familiarity. By combining single-unit recordings and analysis of neural trajectory decoding, we demonstrated the dynamics of anterior cingulate cortex (ACC) neural activity changes synchronously with the harm avoidance performance of mice. In addition, ACC neurons projected to the mediodorsal thalamus (MDL) to modulate the harm avoidance behavior. Optogenetic activation of the ACC-MDL circuit during non-preferred lever pressing (nPLP) and inhibition of this circuit during preferred lever pressing (PLP) both resulted in the loss of harm avoidance ability. This study revealed the ACC-MDL circuit modulates prosocial behavior to avoid harm to conspecifics and may shed light on the treatment of neuropsychiatric disorders with dysfunction of prosocial behavior.

Natural scene reconstruction from fMRI signals using generative latent diffusion

In neural decoding research, one of the most intriguing topics is the reconstruction of perceived natural images based on fMRI signals. Previous studies have succeeded in re-creating different aspects of the visuals, such as low-level properties (shape, texture, layout) or high-level features (category of objects, descriptive semantics of scenes) but have typically failed to reconstruct these properties together for complex scene images. Generative AI has recently made a leap forward with latent diffusion models capable of generating high-complexity images. Here, we investigate how to take advantage of this innovative technology for brain decoding. We present a two-stage scene reconstruction framework called “Brain-Diffuser”. In the first stage, starting from fMRI signals, we reconstruct images that capture low-level properties and overall layout using a VDVAE (Very Deep Variational Autoencoder) model. In the second stage, we use the image-to-image framework of a latent diffusion model (Versatile Diffusion) conditioned on predicted multimodal (text and visual) features, to generate final reconstructed images. On the publicly available Natural Scenes Dataset benchmark, our method outperforms previous models both qualitatively and quantitatively. When applied to synthetic fMRI patterns generated from individual ROI (region-of-interest) masks, our trained model creates compelling “ROI-optimal” scenes consistent with neuroscientific knowledge. Thus, the proposed methodology can have an impact on both applied (e.g. brain–computer interface) and fundamental neuroscience.

Information-Theoretic Neural Decoding Reproduces Several Laws of Human Behavior.

Human response times conform to several regularities including the Hick-Hyman law, the power law of practice, speed-accuracy trade-offs, and the Stroop effect. Each of these has been thoroughly modeled in isolation, but no account describes these phenomena as predictions of a unified framework. We provide such a framework and show that the phenomena arise as decoding times in a simple neural rate code with an entropy stopping threshold. Whereas traditional information-theoretic encoding systems exploit task statistics to optimize encoding strategies, we move this optimization to the decoder, treating it as a Bayesian ideal observer that can track transmission statistics as prior information during decoding. Our approach allays prominent concerns that applying information-theoretic perspectives to modeling brain and behavior requires complex encoding schemes that are incommensurate with neural encoding.

Neural layered min-sum decoders for cyclic codes

This paper proposes a low-complexity neural network decoder based on the layered min-sum algorithm to decode cyclic codes. By generalizing the layered min-sum algorithm to its neural network counterpart, the number of network weights decreases while retaining a good error correction performance. The Bose–Chaudhuri–Hocquenghem (BCH) codes, quadratic residue (QR) codes, and punctured Reed–Muller (RM) codes are selected as three exemplary binary cyclic codes. Simulation results show that the proposed neural decoder achieves superior performance with less computational complexity compared with the state-of-the-art neural network decoder. Further, a neural decoder incorporating the modified random redundant decoding (mRRD) algorithm is investigated to approach the performance of maximum-likelihood decoding for some short codes.

A temporal convolutional recurrent autoencoder based framework for compressing time series data

The sharply growing volume of time series data due to recent sensing technology advancement poses emerging challenges to the data transfer speed and storage as well as corresponding energy consumption. To tackle the overwhelming volume of time series data in transmission and storage, compressing time series, which encodes time series into smaller size representations while enables authentic restoration of compressed ones with minimizing the reconstruction error, has attracted significant attention. Numerous methods have been developed and recent deep learning ones with minimal assumptions on data characteristics, such as recurrent autoencoders, have shown themselves to be competitive. Yet, capturing long-term dependencies in time series compression is a significant challenge calling further development. To make a response, this paper proposes a temporal convolutional recurrent autoencoder framework for more effective time series compression. First, two autoencoder modules, the temporal convolutional network encoder with a recurrent neural network decoder (TCN-RNN) and the temporal convolutional network encoder with an attention assisted recurrent neural network decoder (TCN-ARNN), are developed. The TCN-RNN employs only the recurrent neural network decoder to reconstruct the time series in reverse order. In contrast, the TCN-ARNN uses two recurrent neural networks to reconstruct the time series in both forward and reverse order in parallel. In addition, a timestep-wise attention network is developed to incorporate the forward and reverse reconstructions into the ultimate reconstruction with adaptive weights. Finally, a model selection procedure is developed to adaptively select between the TCN-RNN and TCN-ARNN based on their reconstruction performance on the validation dataset. Computational experiments on five datasets show that the proposed temporal convolutional recurrent autoencoder outperforms state-of-the-art benchmarking models in terms of lower reconstruction errors with the same compression ratio, achieving an improvement of up to 45.14% in the average of mean squared errors. Results indicate a promising potential of the proposed temporal convolutional recurrent autoencoder on the time series compression for various applications involving long time series data.

Neural decoding reveals specialized kinematic tuning after an abrupt cortical transition

The primary motor cortex (M1) exhibits a protracted period of development, including the development of a sensory representation long before motor outflow emerges. In rats, this representation is present by postnatal day (P) 8, when M1 activity is "discontinuous." Here, we ask how the representation changes upon the transition to "continuous" activity at P12. We use neural decoding to predict forelimb movements from M1 activity and show that a linear decoder effectively predicts limb movements at P8 but not at P12; instead, a nonlinear decoder better predicts limb movements at P12. The altered decoder performance reflects increased complexity and uniqueness of kinematic information in M1. We next show that M1's representation at P12 is more susceptible to "lesioning" of inputs and "transplanting" of M1's encoding scheme from one pup to another. Thus, the emergence of continuous M1 activity signals the developmental onset of more complex, informationally sparse, and individualized sensory representations.

Evaluating Performances of Attention-Based Merge Architecture Models for Image Captioning in Indian Languages

Image captioning is a growing topic of research in which numerous advancements have been made in the past few years. Deep learning methods have been used extensively for generating textual descriptions of image data. In addition, attention-based image captioning mechanisms have also been proposed, which give state-ofthe- art results in image captioning. However, many applications and analyses of these methodologies have not been made in the case of languages from the Indian subcontinent. This paper presents attention-based merge architecture models to achieve accurate captions of images in four Indian languages- Marathi, Kannada, Malayalam, and Tamil. The widely known Flickr8K dataset was used for this project. Pre-trained Convolutional Neural Network (CNN) models and language decoder attention models were implemented, which serve as the components of the mergearchitecture proposed here. Finally, the accuracy of the generated captions was compared against the gold captions using Bilingual Evaluation Understudy (BLEU) as an evaluation metric. It was observed that the merge architectures consisting of InceptionV3 give the best results for the languages we test on, the scores discussed in the paper. Highest BLEU-1 scores obtained for each language were: 0.4939 for Marathi, 0.4557 for Kannada, 0.5082 for Malayalam, and 0.5201 for Tamil. Our proposed architectures gave much higher scores than other architectures implemented for these languages.

A channel estimation method using denoising autoencoder for large-scale asymmetric backscatter systems

A novel channel estimation method based on deep learning algorithm is proposed for large-scale IoT networks. We consider asymmetric backscatter communication system to maintain low-power at sensor nodes. In order to obtain channel data, we design denoising autoencoder which consists of encoder with Feedforward Neural Network (FNN) and decoder with Convolutional Neural Network (CNN). Finally, the channel estimation error is minimized, while the pilots are optimized. Especially, we adopt beamforming technique that relies only on cascaded channel data to reduce complexity in multi-sensor system. It is shown that the accuracy is slightly degraded while the complexity is greatly reduced.

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features.

Decoding human visual neural representations is a challenging task with great scientific significance in revealing vision-processing mechanisms and developing brain-like intelligent machines. Most existing methods are difficult to generalize to novel categories that have no corresponding neural data for training. The two main reasons are 1) the under-exploitation of the multimodal semantic knowledge underlying the neural data and 2) the small number of paired (stimuli-responses) training data. To overcome these limitations, this paper presents a generic neural decoding method called BraVL that uses multimodal learning of brain-visual-linguistic features. We focus on modeling the relationships between brain, visual and linguistic features via multimodal deep generative models. Specifically, we leverage the mixture-of-product-of-experts formulation to infer a latent code that enables a coherent joint generation of all three modalities. To learn a more consistent joint representation and improve the data efficiency in the case of limited brain activity data, we exploit both intra- and inter-modality mutual information maximization regularization terms. In particular, our BraVL model can be trained under various semi-supervised scenarios to incorporate the visual and textual features obtained from the extra categories. Finally, we construct three trimodal matching datasets, and the extensive experiments lead to some interesting conclusions and cognitive insights: 1) decoding novel visual categories from human brain activity is practically possible with good accuracy; 2) decoding models using the combination of visual and linguistic features perform much better than those using either of them alone; 3) visual perception may be accompanied by linguistic influences to represent the semantics of visual stimuli.

Sentence-Level Neural Language Decoding based on Speech Imagery from EEG Signals

Sentence-Level Neural Language Decoding based on Speech Imagery from EEG Signals

Enhancing Prediction of Forelimb Movement Trajectory through a Calibrating-Feedback Paradigm Incorporating RAT Primary Motor and Agranular Cortical Ensemble Activity in the Goal-Directed Reaching Task.

Complete reaching movements involve target sensing, motor planning, and arm movement execution, and this process requires the integration and communication of various brain regions. Previously, reaching movements have been decoded successfully from the motor cortex (M1) and applied to prosthetic control. However, most studies attempted to decode neural activities from a single brain region, resulting in reduced decoding accuracy during visually guided reaching motions. To enhance the decoding accuracy of visually guided forelimb reaching movements, we propose a parallel computing neural network using both M1 and medial agranular cortex (AGm) neural activities of rats to predict forelimb-reaching movements. The proposed network decodes M1 neural activities into the primary components of the forelimb movement and decodes AGm neural activities into internal feedforward information to calibrate the forelimb movement in a goal-reaching movement. We demonstrate that using AGm neural activity to calibrate M1 predicted forelimb movement can improve decoding performance significantly compared to neural decoders without calibration. We also show that the M1 and AGm neural activities contribute to controlling forelimb movement during goal-reaching movements, and we report an increase in the power of the local field potential (LFP) in beta and gamma bands over AGm in response to a change in the target distance, which may involve sensorimotor transformation and communication between the visual cortex and AGm when preparing for an upcoming reaching movement. The proposed parallel computing neural network with the internal feedback model improves prediction accuracy for goal-reaching movements.

Zero-Shot Neural Decoding with Semi-Supervised Multi-View Embedding.

Zero-shot neural decoding aims to decode image categories, which were not previously trained, from functional magnetic resonance imaging (fMRI) activity evoked when a person views images. However, having insufficient training data due to the difficulty in collecting fMRI data causes poor generalization capability. Thus, models suffer from the projection domain shift problem when novel target categories are decoded. In this paper, we propose a zero-shot neural decoding approach with semi-supervised multi-view embedding. We introduce the semi-supervised approach that utilizes additional images related to the target categories without fMRI activity patterns. Furthermore, we project fMRI activity patterns into a multi-view embedding space, i.e., visual and semantic feature spaces of viewed images to effectively exploit the complementary information. We define several source and target groups whose image categories are very different and verify the zero-shot neural decoding performance. The experimental results demonstrate that the proposed approach rectifies the projection domain shift problem and outperforms existing methods.

Machine Learning-Assisted Watermarking via Bias Control of Electro-Optical Modulator

We propose an optical watermarking strategy based on precise bias tuning of an electro-optical modulator and a convolutional neural network-enabled decoder. This scheme can be used for identity authentication at the physical layer of optical networks. This method is demonstrated with a transmission demo of about 250 kbit/s watermarking data in a 10-Gbit/s public 20-km communication. The watermarking data is modulated on the bias states of the modulator based on the precise and stable bias control technique. The watermark embedding has been verified to make almost no impact on the 20-km public transmission. For the legal receiver, the invisible watermark data can be extracted with the aid of the convolutional neural network-enabled decoder to verify the legality of the received data. Only one optical source is needed in the proposed scheme and the watermarking is integrated into the public signal naturally. We analyzed and verified the influence of watermarking data on the bit error rate and the channel capacity of the public data. The privacy and security of watermarking data are also analyzed.

Graph-Enhanced Emotion Neural Decoding.

Brain signal-based emotion recognition has recently attracted considerable attention since it has powerful potential to be applied in human-computer interaction. To realize the emotional interaction of intelligent systems with humans, researchers have made efforts to decode human emotions from brain imaging data. The majority of current efforts use emotion similarities (e.g., emotion graphs) or brain region similarities (e.g., brain networks) to learn emotion and brain representations. However, the relationships between emotions and brain regions are not explicitly incorporated into the representation learning process. As a result, the learned representations may not be informative enough to benefit specific tasks, e.g., emotion decoding. In this work, we propose a novel idea of graph-enhanced emotion neural decoding, which takes advantage of a bipartite graph structure to integrate the relationships between emotions and brain regions into the neural decoding process, thus helping learn better representations. Theoretical analyses conclude that the suggested emotion-brain bipartite graph inherits and generalizes the conventional emotion graphs and brain networks. Comprehensive experiments on visually evoked emotion datasets demonstrate the effectiveness and superiority of our approach.

Jump-GRS: a multi-phase approach to structured pruning of neural networks for neural decoding

Objective. Neural decoding, an important area of neural engineering, helps to link neural activity to behavior. Deep neural networks (DNNs), which are becoming increasingly popular in many application fields of machine learning, show promising performance in neural decoding compared to traditional neural decoding methods. Various neural decoding applications, such as brain computer interface applications, require both high decoding accuracy and real-time decoding speed. Pruning methods are used to produce compact DNN models for faster computational speed. Greedy inter-layer order with Random Selection (GRS) is a recently-designed structured pruning method that derives compact DNN models for calcium-imaging-based neural decoding. Although GRS has advantages in terms of detailed structure analysis and consideration of both learned information and model structure during the pruning process, the method is very computationally intensive, and is not feasible when large-scale DNN models need to be pruned within typical constraints on time and computational resources. Large-scale DNN models arise in neural decoding when large numbers of neurons are involved. In this paper, we build on GRS to develop a new structured pruning algorithm called jump GRS (JGRS) that is designed to efficiently compress large-scale DNN models. Approach. On top of GRS, JGRS implements a ‘jump mechanism’, which bypasses retraining intermediate models when model accuracy is relatively less sensitive to pruning operations. Design of the jump mechanism is motivated by identifying different phases of the structured pruning process, where retraining can be done infrequently in earlier phases without sacrificing accuracy. The jump mechanism helps to significantly speed up execution of the pruning process and greatly enhance its scalability. We compare the pruning performance and speed of JGRS and GRS with extensive experiments in the context of neural decoding. Main results. Our results demonstrate that JGRS provides significantly faster pruning speed compared to GRS, and at the same time, JGRS provides pruned models that are similarly compact as those generated by GRS. Significance. In our experiments, we demonstrate that JGRS achieves on average 9%–20% more compressed models compared to GRS with 2–8 times faster speed (less time required for pruning) across four different initial models on a relevant dataset for neural data analysis.

Reconstructing controllable faces from brain activity with hierarchical multiview representations

Reconstructing visual experience from brain responses measured by functional magnetic resonance imaging (fMRI) is a challenging yet important research topic in brain decoding, especially it has proved more difficult to decode visually similar stimuli, such as faces. Although face attributes are known as the key to face recognition, most existing methods generally ignore how to decode facial attributes more precisely in perceived face reconstruction, which often leads to indistinguishable reconstructed faces. To solve this problem, we propose a novel neural decoding framework called VSPnet (voxel2style2pixel) by establishing hierarchical encoding and decoding networks with disentangled latent representations as media, so that to recover visual stimuli more elaborately. And we design a hierarchical visual encoder (named HVE) to pre-extract features containing both high-level semantic knowledge and low-level visual details from stimuli. The proposed VSPnet consists of two networks: Multi-branch cognitive encoder and style-based image generator. The encoder network is constructed by multiple linear regression branches to map brain signals to the latent space provided by the pre-extracted visual features and obtain representations containing hierarchical information consistent to the corresponding stimuli. We make the generator network inspired by StyleGAN to untangle the complexity of fMRI representations and generate images. And the HVE network is composed of a standard feature pyramid over a ResNet backbone. Extensive experimental results on the latest public datasets have demonstrated the reconstruction accuracy of our proposed method outperforms the state-of-the-art approaches and the identifiability of different reconstructed faces has been greatly improved. In particular, we achieve feature editing for several facial attributes in fMRI domain based on the multiview (i.e., visual stimuli and evoked fMRI) latent representations.

A scalable and fast artificial neural network syndrome decoder for surface codes

Surface code error correction offers a highly promising pathway to achieve scalable fault-tolerant quantum computing. When operated as stabilizer codes, surface code computations consist of a syndrome decoding step where measured stabilizer operators are used to determine appropriate corrections for errors in physical qubits. Decoding algorithms have undergone substantial development, with recent work incorporating machine learning (ML) techniques. Despite promising initial results, the ML-based syndrome decoders are still limited to small scale demonstrations with low latency and are incapable of handling surface codes with boundary conditions and various shapes needed for lattice surgery and braiding. Here, we report the development of an artificial neural network (ANN) based scalable and fast syndrome decoder capable of decoding surface codes of arbitrary shape and size with data qubits suffering from the depolarizing error model. Based on rigorous training over 50 million random quantum error instances, our ANN decoder is shown to work with code distances exceeding 1000 (more than 4 million physical qubits), which is the largest ML-based decoder demonstration to-date. The established ANN decoder demonstrates an execution time in principle independent of code distance, implying that its implementation on dedicated hardware could potentially offer surface code decoding times of O(μsec), commensurate with the experimentally realisable qubit coherence times. With the anticipated scale-up of quantum processors within the next decade, their augmentation with a fast and scalable syndrome decoder such as developed in our work is expected to play a decisive role towards experimental implementation of fault-tolerant quantum information processing.

Unsupervised neural decoding of signals recorded by thin-film electrode arrays implanted in muscles using autoencoding with a physiologically derived optimisation criterion

We present an autoencoder that learns the discharge times of individual motor units (MU) by processing multichannel electromyogram (EMG) recordings obtained by electrode arrays implanted in muscles, thereby providing a neural interface with the spinal cord. To this end, after preprocessing of the EMG via convolutive sphering, the encoder is constrained to apply an orthogonal transformation to the spatial data and the optimisation criterion enforces a temporal sparsity constraint. These constraints are based on the theoretical modelling of EMG generation. This decomposition method was evaluated on both simulated and experimental signals. For simulated signals, with a ratio of 1.5 between observations and sources, the average detection accuracy was 94% for up to 60 sources for SNR 20 dB. Moreover, the detection accuracy did not significantly decrease when decreasing the SNR to values as low as 0 dB. Experimental signals were collected in humans with thin-film invasive electrodes implanted in the tibialis anterior muscle. The results showed an accuracy on experimental data >90%. Moreover, the proposed method outperformed a state-of-the-art blind source separation approach in terms of the number of reliably detected motor units. In conclusion, we have proposed the first fully unsupervised neural network approach to the problem of neural decoding of intramuscular EMG time series by translating the available theoretical knowledge on the signal properties into an autoencoding architecture tailored to the decomposition problem.