Nonlinear Decoding Research Articles

Revealing unexpected complex encoding but simple decoding mechanisms in motor cortex via separating behaviorally relevant neural signals.

In motor cortex, behaviorally relevant neural responses are entangled with irrelevant signals, which complicates the study of encoding and decoding mechanisms. It remains unclear whether behaviorally irrelevant signals could conceal some critical truth. One solution is to accurately separate behaviorally relevant and irrelevant signals at both single-neuron and single-trial levels, but this approach remains elusive due to the unknown ground truth of behaviorally relevant signals. Therefore, we propose a framework to define, extract, and validate behaviorally relevant signals. Analyzing separated signals in three monkeys performing different reaching tasks, we found neural responses previously considered to contain little information actually encode rich behavioral information in complex nonlinear ways. These responses are critical for neuronal redundancy and reveal movement behaviors occupy a higher-dimensional neural space than previously expected. Surprisingly, when incorporating often-ignored neural dimensions, behaviorally relevant signals can be decoded linearly with comparable performance to nonlinear decoding, suggesting linear readout may be performed in motor cortex. Our findings prompt that separating behaviorally relevant signals may help uncover more hidden cortical mechanisms.

Compact Key Function Secret Sharing with Non-linear Decoder

Compact Key Function Secret Sharing with Non-linear Decoder

High-resolution neural recordings improve the accuracy of speech decoding

Patients suffering from debilitating neurodegenerative diseases often lose the ability to communicate, detrimentally affecting their quality of life. One solution to restore communication is to decode signals directly from the brain to enable neural speech prostheses. However, decoding has been limited by coarse neural recordings which inadequately capture the rich spatio-temporal structure of human brain signals. To resolve this limitation, we performed high-resolution, micro-electrocorticographic (µECoG) neural recordings during intra-operative speech production. We obtained neural signals with 57× higher spatial resolution and 48% higher signal-to-noise ratio compared to macro-ECoG and SEEG. This increased signal quality improved decoding by 35% compared to standard intracranial signals. Accurate decoding was dependent on the high-spatial resolution of the neural interface. Non-linear decoding models designed to utilize enhanced spatio-temporal neural information produced better results than linear techniques. We show that high-density µECoG can enable high-quality speech decoding for future neural speech prostheses.

Improving reduced-order models through nonlinear decoding of projection-dependent outputs

A fundamental hindrance to building data-driven reduced-order models (ROMs) is the poor topological quality of a low-dimensional data projection. This includes behavior such as overlapping, twisting, or large curvatures or uneven data density that can generate nonuniqueness and steep gradients in quantities of interest (QoIs). Here, we employ an encoder-decoder neural network architecture for dimensionality reduction. We find that nonlinear decoding of projection-dependent QoIs, when embedded in a dimensionality reduction technique, promotes improved low-dimensional representations of complex multiscale and multiphysics datasets. When data projection (encoding) is affected by forcing accurate nonlinear reconstruction of the QoIs (decoding), we minimize nonuniqueness and gradients in representing QoIs on a projection. This in turn leads to enhanced predictive accuracy of a ROM. Our findings are relevant to a variety of disciplines that develop data-driven ROMs of dynamical systems such as reacting flows, plasma physics, atmospheric physics, or computational neuroscience.

Neural decoding reveals specialized kinematic tuning after an abrupt cortical transition

The primary motor cortex (M1) exhibits a protracted period of development, including the development of a sensory representation long before motor outflow emerges. In rats, this representation is present by postnatal day (P) 8, when M1 activity is "discontinuous." Here, we ask how the representation changes upon the transition to "continuous" activity at P12. We use neural decoding to predict forelimb movements from M1 activity and show that a linear decoder effectively predicts limb movements at P8 but not at P12; instead, a nonlinear decoder better predicts limb movements at P12. The altered decoder performance reflects increased complexity and uniqueness of kinematic information in M1. We next show that M1's representation at P12 is more susceptible to "lesioning" of inputs and "transplanting" of M1's encoding scheme from one pup to another. Thus, the emergence of continuous M1 activity signals the developmental onset of more complex, informationally sparse, and individualized sensory representations.

Music can be reconstructed from human auditory cortex activity using nonlinear decoding models.

Music is core to human experience, yet the precise neural dynamics underlying music perception remain unknown. We analyzed a unique intracranial electroencephalography (iEEG) dataset of 29 patients who listened to a Pink Floyd song and applied a stimulus reconstruction approach previously used in the speech domain. We successfully reconstructed a recognizable song from direct neural recordings and quantified the impact of different factors on decoding accuracy. Combining encoding and decoding analyses, we found a right-hemisphere dominance for music perception with a primary role of the superior temporal gyrus (STG), evidenced a new STG subregion tuned to musical rhythm, and defined an anterior-posterior STG organization exhibiting sustained and onset responses to musical elements. Our findings show the feasibility of applying predictive modeling on short datasets acquired in single patients, paving the way for adding musical elements to brain-computer interface (BCI) applications.

Reservoir based spiking models for univariate Time Series Classification.

A variety of advanced machine learning and deep learning algorithms achieve state-of-the-art performance on various temporal processing tasks. However, these methods are heavily energy inefficient-they run mainly on the power hungry CPUs and GPUs. Computing with Spiking Networks, on the other hand, has shown to be energy efficient on specialized neuromorphic hardware, e.g., Loihi, TrueNorth, SpiNNaker, etc. In this work, we present two architectures of spiking models, inspired from the theory of Reservoir Computing and Legendre Memory Units, for the Time Series Classification (TSC) task. Our first spiking architecture is closer to the general Reservoir Computing architecture and we successfully deploy it on Loihi; the second spiking architecture differs from the first by the inclusion of non-linearity in the readout layer. Our second model (trained with Surrogate Gradient Descent method) shows that non-linear decoding of the linearly extracted temporal features through spiking neurons not only achieves promising results, but also offers low computation-overhead by significantly reducing the number of neurons compared to the popular LSM based models-more than 40x reduction with respect to the recent spiking model we compare with. We experiment on five TSC datasets and achieve new SoTA spiking results (-as much as 28.607% accuracy improvement on one of the datasets), thereby showing the potential of our models to address the TSC tasks in a green energy-efficient manner. In addition, we also do energy profiling and comparison on Loihi and CPU to support our claims.

Nonlinear dynamic mode decomposition from time-resolving snapshots based on deep convolutional autoencoder

A novel nonlinear decomposition method of flow field is proposed based on the dynamic mode decomposition (DMD) and nonlinear mode decomposition autoencoder. The flow fields are indexed in time order and then input to the nonlinear neural networks to learn the connected observables and decomposed fields. The reconstructions of input fields are assumed to be the summation of the nonlinear decomposed fields. The nonlinear decoders are regarded as the nonlinear modes, which concentrate most of the total energy of the flow field data. The resulting nonlinear dynamic mode decomposition autoencoder reports a series of orderly low-dimensional representations and decomposition fields. Besides, the proposed method can be used for dynamic modeling and returns more stable and accurate predictions with a few number of low-dimensional representations. The present method is tested with the benchmark case, flow around a circular cylinder at Reynolds number Re = 100. The results in this example indicate that the proposed method achieves higher reconstruction accuracy using fewer modes, while retaining similar temporal dynamics and mode information as proper orthogonal decomposition and DMD.

Intracortical Hindlimb Brain–Computer Interface Systems: A Systematic Review

Brain-computer interfaces (BCI) can help people with motor disorders to regain their ability to communicate and interact with the surrounding environment. The majority of studies in this field pursue the development of BCI systems to enhance or restore the movement functionality of people with disability. Although the studies on the development of BCIs to restore hindlimb movements have shorter backgrounds compared to forelimb, several studies have investigated hindlimb BCIs and their results were promising. In the present study, we systematically reviewed the studies investigating the decoding of hindlimb movement parameters using intracortical signals. Three scientific databases (PubMed, Scopus, and Embase) were used to extract the articles and the experiment, recording, processing methods, and results of the included studies were discussed. Although several studies on upper-limb intracortical BCIs have been conducted on human subjects, almost all studies in hindlimb intracortical BCIs field were performed on animal subjects. The most investigated task was walking on a treadmill, and the position of hindlimb joints and gait phase were the most studied continuous and discrete parameters, respectively. The included studies have mainly used spikes and linear decoders, which leaves the question of the effectiveness of using local field potentials and nonlinear decoders in this field unanswered. Although the results imply that hindlimb movement decoding using brain signals is feasible in laboratory conditions, further investigations are required to examine the hindlimb BCIs in real-life conditions.

Latent adversarial regularized autoencoder for high-dimensional probabilistic time series prediction

Many practical applications require probabilistic prediction of time series to model the distribution on future horizons. With ever-increasing dimensions, much effort has been invested into developing methods that often make an assumption about the independence between time series. Consequently, the probabilistic prediction in high-dimensional environments has become an essential topic with significant challenges. In this paper, we propose a novel probabilistic model called latent adversarial regularized autoencoder, abbreviated as TimeLAR, specifically for high-dimensional multivariate Time Series Prediction (TSP). It integrates the flexibility of Generative Adversarial Networks (GANs) and the capability of autoencoders in extracting higher-level non-linear features. Through flexible autoencoder mapping, TimeLAR learns cross-series relationships and encodes this global information into several latent variables. We design a modified Transformer for these latent variables to capture global temporal patterns and infer latent space prediction distributions, where only one step is required to output multi-step predictions. Furthermore, we employ the GAN to further refine the performance of latent space predictions, by using a discriminator to guide the training of the autoencoder and the Transformer in an adversarial process. Finally, complex distributions of multivariate time series data can be modeled by the non-linear decoder of the autoencoder. The effectiveness of TimeLAR is empirically underpinned by extensive experiments conducted on five real-world high-dimensional time series datasets in the fields of transportation, electricity, and web page views.

Extrinsic crosstalk suppression based on a calibration convolutional neural network for glasses-free 3D display

Extrinsic crosstalk degrades the 3D imaging quality and it occurs due to fabrication errors of the light control optical component and assembling errors of the display system. Here, an extrinsic crosstalk suppression method based on a calibration convolutional neural network (C-CNN) is demonstrated for glasses-free 3D display. The C-CNN is established to calibrate the 3D imaging with extrinsic crosstalk, which can obtain the high-order nonlinear decoding function of the light control optical component with fabrication errors and assembling errors. Furthermore, model for 3D imaging with both extrinsic and intrinsic crosstalk can be mathematically established using the C-CNN integrated with the frequency response calculation. Additionally, according to the calibration for real-life 3D imaging with extrinsic crosstalk, pixels of elemental image array (EIA) contributing to viewpoint crosstalk are extinguished to suppress extrinsic crosstalk. The experiment verifies the feasibility and superior of the proposed method. A 5-degree viewing angle and 5 viewpoints glasses-free 3D display with improved 3D imaging quality is demonstrated.

Revealing nonlinear neural decoding by analyzing choices

Sensory data about most natural task-relevant variables are entangled with task-irrelevant nuisance variables. The neurons that encode these relevant signals typically constitute a nonlinear population code. Here we present a theoretical framework for quantifying how the brain uses or decodes its nonlinear information. Our theory obeys fundamental mathematical limitations on information content inherited from the sensory periphery, describing redundant codes when there are many more cortical neurons than primary sensory neurons. The theory predicts that if the brain uses its nonlinear population codes optimally, then more informative patterns should be more correlated with choices. More specifically, the theory predicts a simple, easily computed quantitative relationship between fluctuating neural activity and behavioral choices that reveals the decoding efficiency. This relationship holds for optimal feedforward networks of modest complexity, when experiments are performed under natural nuisance variation. We analyze recordings from primary visual cortex of monkeys discriminating the distribution from which oriented stimuli were drawn, and find these data are consistent with the hypothesis of near-optimal nonlinear decoding.

Latent Linear Adjustment Autoencoder v1.0: a novel method for estimating and emulating dynamic precipitation at high resolution

Abstract. A key challenge in climate science is to quantify the forced response in impact-relevant variables such as precipitation against the background of internal variability, both in models and observations. Dynamical adjustment techniques aim to remove unforced variability from a target variable by identifying patterns associated with circulation, thus effectively acting as a filter for dynamically induced variability. The forced contributions are interpreted as the variation that is unexplained by circulation. However, dynamical adjustment of precipitation at local scales remains challenging because of large natural variability and the complex, nonlinear relationship between precipitation and circulation particularly in heterogeneous terrain. Building on variational autoencoders, we introduce a novel statistical model – the Latent Linear Adjustment Autoencoder (LLAAE) – that enables estimation of the contribution of a coarse-scale atmospheric circulation proxy to daily precipitation at high resolution and in a spatially coherent manner. To predict circulation-induced precipitation, the Latent Linear Adjustment Autoencoder combines a linear component, which models the relationship between circulation and the latent space of an autoencoder, with the autoencoder's nonlinear decoder. The combination is achieved by imposing an additional penalty in the cost function that encourages linearity between the circulation field and the autoencoder's latent space, hence leveraging robustness advantages of linear models as well as the flexibility of deep neural networks. We show that our model predicts realistic daily winter precipitation fields at high resolution based on a 50-member ensemble of the Canadian Regional Climate Model at 12 km resolution over Europe, capturing, for instance, key orographic features and geographical gradients. Using the Latent Linear Adjustment Autoencoder to remove the dynamic component of precipitation variability, forced thermodynamic components are expected to remain in the residual, which enables the uncovering of forced precipitation patterns of change from just a few ensemble members. We extend this to quantify the forced pattern of change conditional on specific circulation regimes. Future applications could include, for instance, weather generators emulating climate model simulations of regional precipitation, detection and attribution at subcontinental scales, or statistical downscaling and transfer learning between models and observations to exploit the typically much larger sample size in models compared to observations.

Auditory attention decoding from electroencephalography based on long short-term memory networks

Available researches have confirmed that the target speech stream of human listener's selective attention in a multi-speaker scenario can be decoded from ongoing electroencephalography (EEG) signals. Due to its great potential application for the neuro-steered hearing devices, auditory attention decoding has recently received increasing attention. In practice, development of high-accuracy decoding performance and use of EEG signals with as short duration as possible are urgently needed for auditory attention decoding in real-time scenarios. However, these are still great challenges for the existing decoding methods to achieve such performances. In this paper, we propose a nonlinear modeling approach between EEG signals and speech envelopes by exploiting long short-term memory networks (LSTM), a special structure of recurrent neural networks (RNN), to decipher selective auditory attention from single-trial EEG signals, which is capable of learning long-term dependencies between EEG responses and continuous auditory stimuli. Unlike the existing linear modeling approaches, the LSTM RNN-based auditory attention decoding method is an end-to-end nonlinear decoding framework that subsumes the explicit similarity determination as part of feature learning in deep neural networks. The proposed decoding method is evaluated using the real EEG data collected from 21 subjects. Experimental results show that our proposed decoding method can achieve the state-of-the-art performance to identify the target speech in a competing two-talker scenario. When used the EEG signals with 1 s duration, the average accuracy for the 21 subjects' attention decoding is reached to 96.12%. The results show great potentials for the real-time auditory attention detection in neuro-steered hearing aids.

Nonlinear Decoding of Natural Images From Large-Scale Primate Retinal Ganglion Recordings.

Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve on existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from more than 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations for accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons.

Population Coding of Natural Electrosensory Stimuli by Midbrain Neurons.

Natural stimuli display spatiotemporal characteristics that typically vary over orders of magnitude, and their encoding by sensory neurons remains poorly understood. We investigated population coding of highly heterogeneous natural electrocommunication stimuli in Apteronotus leptorhynchus of either sex. Neuronal activities were positively correlated with one another in the absence of stimulation, and correlation magnitude decayed with increasing distance between recording sites. Under stimulation, we found that correlations between trial-averaged neuronal responses (i.e., signal correlations) were positive and higher in magnitude for neurons located close to another, but that correlations between the trial-to-trial variability (i.e., noise correlations) were independent of physical distance. Overall, signal and noise correlations were independent of stimulus waveform as well as of one another. To investigate how neuronal populations encoded natural electrocommunication stimuli, we considered a nonlinear decoder for which the activities were combined. Decoding performance was best for a timescale of 6 ms, indicating that midbrain neurons transmit information via precise spike timing. A simple summation of neuronal activities (equally weighted sum) revealed that noise correlations limited decoding performance by introducing redundancy. Using an evolution algorithm to optimize performance when considering instead unequally weighted sums of neuronal activities revealed much greater performance values, indicating that midbrain neuron populations transmit information that reliably enable discrimination between different stimulus waveforms. Interestingly, we found that different weight combinations gave rise to similar discriminability, suggesting robustness. Our results have important implications for understanding how natural stimuli are integrated by downstream brain areas to give rise to behavioral responses.SIGNIFICANCE STATEMENT We show that midbrain electrosensory neurons display correlations between their activities and that these can significantly impact performance of decoders. While noise correlations limited discrimination performance by introducing redundancy, considering unequally weighted sums of neuronal activities gave rise to much improved performance and mitigated the deleterious effects of noise correlations. Further analysis revealed that increased discriminability was achieved by making trial-averaged responses more separable, as well as by reducing trial-to-trial variability by eliminating noise correlations. We further found that multiple combinations of weights could give rise to similar discrimination performances, which suggests that such combinatorial codes could be achieved in the brain. We conclude that the activities of midbrain neuronal populations can be used to reliably discriminate between highly heterogeneous stimulus waveforms.

Decoding Kinematic Information From Primary Motor Cortex Ensemble Activities Using a Deep Canonical Correlation Analysis.

The control of arm movements through intracortical brain–machine interfaces (BMIs) mainly relies on the activities of the primary motor cortex (M1) neurons and mathematical models that decode their activities. Recent research on decoding process attempts to not only improve the performance but also simultaneously understand neural and behavioral relationships. In this study, we propose an efficient decoding algorithm using a deep canonical correlation analysis (DCCA), which maximizes correlations between canonical variables with the non-linear approximation of mappings from neuronal to canonical variables via deep learning. We investigate the effectiveness of using DCCA for finding a relationship between M1 activities and kinematic information when non-human primates performed a reaching task with one arm. Then, we examine whether using neural activity representations from DCCA improves the decoding performance through linear and non-linear decoders: a linear Kalman filter (LKF) and a long short-term memory in recurrent neural networks (LSTM-RNN). We found that neural representations of M1 activities estimated by DCCA resulted in more accurate decoding of velocity than those estimated by linear canonical correlation analysis, principal component analysis, factor analysis, and linear dynamical system. Decoding with DCCA yielded better performance than decoding the original FRs using LSTM-RNN (6.6 and 16.0% improvement on average for each velocity and position, respectively; Wilcoxon rank sum test, p < 0.05). Thus, DCCA can identify the kinematics-related canonical variables of M1 activities, thus improving the decoding performance. Our results may help advance the design of decoding models for intracortical BMIs.

Non-linear online low-frequency EEG decoding of arm movements during a pursuit tracking task.

Decoding upper-limb movements in invasive recordings has become a reality, but neural tuning in non-invasive low-frequency recordings is still under discussion. Recent studies managed to decode movement positions and velocities using linear decoders, even developing an online system. The decoded signals, however, exhibited smaller amplitudes than actual movements, affecting feedback and user experience. Recently, we showed that a non-linear offline decoder can combine directional (e.g., velocity) and non-directional (e.g., speed) information. In this study, it is assessed if the non-linear decoder can be used online to provide real-time feedback. Five healthy subjects were asked to track a moving target by controlling a robotic arm. Initially, the robot was controlled by their right hand; then, the control was gradually switched until it was entirely controlled by the electroencephalogram (EEG). Correlations between actual and decoded movements were generally above chance level. Results suggest that information about speed was also encoded in the EEG, demonstrating that the proposed non-linear decoder is suitable for decoding real-time arm movements.

Learning efficient illumination multiplexing for joint capture of reflectance and shape

We propose a novel framework that automatically learns the lighting patterns for efficient, joint acquisition of unknown reflectance and shape. The core of our framework is a deep neural network, with a shared linear encoder that directly corresponds to the lighting patterns used in physical acquisition, as well as non-linear decoders that output per-pixel normal and diffuse / specular information from photographs. We exploit the diffuse and normal information from multiple views to reconstruct a detailed 3D shape, and then fit BRDF parameters to the diffuse / specular information, producing texture maps as reflectance results. We demonstrate the effectiveness of the framework with physical objects that vary considerably in reflectance and shape, acquired with as few as 16 ~ 32 lighting patterns that correspond to 7 ~ 15 seconds of per-view acquisition time. Our framework is useful for optimizing the efficiency in both novel and existing setups, as it can automatically adapt to various factors, including the geometry / the lighting layout of the device and the properties of appearance.

Towards Intelligent Intracortical BMI (i 2BMI): Low-Power Neuromorphic Decoders That Outperform Kalman Filters.

Fully-implantable wireless intracortical Brain Machine Interfaces (iBMI) is one of the most promising next frontiers in the nascent field of neurotechnology. However, scaling the number of channels in such systems by another 10× is difficult due to power and bandwidth requirements of the wireless transmitter. One promising solution for that is to include more processing, up to the decoder, in the implant so that transmission data-rate is reduced drastically. Earlier work on neuromorphic decoder chips only showed classification of discrete states. We present results for continuous state decoding using a low-power neuromorphic decoder chip termed Spike-input Extreme Learning Machine (SELMA) that implements a nonlinear decoder without memory and its memory-based version with time-delayed bins, SELMA-bins. We have compared SELMA, SELMA-bins against state-of-the-art Steady-State Kalman Filter (SSKF), a linear decoder with memory, across two different datasets involving a total of 4 non-human primates (NHPs). Results show at least a 10% (20%) or more increase in the fraction of variance accounted for (FVAF) by SELMA (SELMA-bins) over SSKF across the datasets. Estimated energy consumption comparison shows SELMA (SELMA-bins) consuming ≈ 9 nJ/update (23 nJ/update) against SSKF's ≈ 7.4 nJ/update for an iBMI with a 10 degree of freedom control. Thus, SELMA yields better performance against SSKF while consuming energy in the same range as SSKF whereas SELMA-bins performs the best with moderately increased energy consumption, albeit far less than energy required for raw data transmission. This paves the way for reducing transmission data rates in future scaled iBMI systems.