Neural Representation Research Articles

Convergent neural signatures of speech prediction error are a biological marker for spoken word recognition

AbstractWe use MEG and fMRI to determine how predictions are combined with speech input in superior temporal cortex. We compare neural responses to words in which first syllables strongly or weakly predict second syllables (e.g., “bingo”, “snigger” versus “tango”, “meagre”). We further compare neural responses to the same second syllables when predictions mismatch with input during pseudoword perception (e.g., “snigo” and “meago”). Neural representations of second syllables are suppressed by strong predictions when predictions match sensory input but show the opposite effect when predictions mismatch. Computational simulations show that this interaction is consistent with prediction error but not alternative (sharpened signal) computations. Neural signatures of prediction error are observed 200 ms after second syllable onset and in early auditory regions (bilateral Heschl’s gyrus and STG). These findings demonstrate prediction error computations during the identification of familiar spoken words and perception of unfamiliar pseudowords.

StegaINR4MIH: steganography by implicit neural representation for multi-image hiding

StegaINR4MIH: steganography by implicit neural representation for multi-image hiding

A combinatorial neural code for long-term motor memory.

Motor skill repertoire can be stably retained over long periods, but the neural mechanism that underlies stable memory storage remains poorly understood1-8. Moreover, it is unknown how existing motor memories are maintained as new motor skills are continuously acquired. Here we tracked neural representation of learned actions throughout a significant portion of the lifespan of a mouse and show that learned actions are stably retained in combination with context, which protects existing memories from erasure during new motor learning. We established a continual learning paradigm in which mice learned to perform directional licking in different task contexts while we tracked motor cortex activity for up to six months using two-photon imaging. Within the same task context, activity driving directional licking was stable over time with little representational drift. When learning new task contexts, new preparatory activity emerged to drive the same licking actions. Learning created parallel new motor memories instead of modifying existing representations. Re-learning to make the same actions in the previous task context re-activated the previous preparatory activity, even months later. Continual learning of new task contexts kept creating new preparatory activity patterns. Context-specific memories, as we observed in the motor system, may provide a solution for stable memory storage throughout continual learning.

Can human brain connectivity explain verbal working memory?

ABSTRACT The ability of humans to store spoken words in verbal working memory and build extensive vocabularies is believed to stem from evolutionary changes in cortical connectivity across primate species. However, the underlying neurobiological mechanisms remain unclear. Why can humans acquire vast vocabularies, while non-human primates cannot? This study addresses this question using brain-constrained neural networks that realize between-species differences in cortical connectivity. It investigates how these structural differences support the formation of neural representations for spoken words and the emergence of verbal working memory, crucial for human vocabulary building. We develop comparative models of frontotemporal and occipital cortices, reflecting human and non-human primate neuroanatomy. Using meanfield and spiking neural networks, we simulate auditory word recognition and examine verbal working memory function. The “human models”, characterized by denser inter-area connectivity in core language areas, produced larger cell assemblies than the “monkey models”, with specific topographies reflecting semantic properties of the represented words. Crucially, longer-lasting reverberant neural activity was observed in human versus monkey architectures, compatible with robust verbal working memory, a necessary condition for vocabulary building. Our findings offer insights into the structural basis of human-specific symbol learning and verbal working memory, shedding light on humans’ unique capacity for large vocabulary acquisition.

NeuralFeels with neural fields: Visuotactile perception for in-hand manipulation.

To achieve human-level dexterity, robots must infer spatial awareness from multimodal sensing to reason over contact interactions. During in-hand manipulation of novel objects, such spatial awareness involves estimating the object's pose and shape. The status quo for in-hand perception primarily uses vision and is restricted to tracking a priori known objects. Moreover, visual occlusion of objects in hand is imminent during manipulation, preventing current systems from pushing beyond tasks without occlusion. We combined vision and touch sensing on a multifingered hand to estimate an object's pose and shape during in-hand manipulation. Our method, NeuralFeels, encodes object geometry by learning a neural field online and jointly tracks it by optimizing a pose graph problem. We studied multimodal in-hand perception in simulation and the real world, interacting with different objects via a proprioception-driven policy. Our experiments showed final reconstruction F scores of 81% and average pose drifts of 4.7 millimeters, which was further reduced to 2.3 millimeters with known object models. In addition, we observed that, under heavy visual occlusion, we could achieve improvements in tracking up to 94% compared with vision-only methods. Our results demonstrate that touch, at the very least, refines and, at the very best, disambiguates visual estimates during in-hand manipulation. We release our evaluation dataset of 70 experiments, FeelSight, as a step toward benchmarking in this domain. Our neural representation driven by multimodal sensing can serve as a perception backbone toward advancing robot dexterity.

Entropy of Neuronal Spike Patterns

Neuronal spike patterns are the fundamental units of neural communication in the brain, which is still not fully understood. Entropy measures offer a quantitative framework to assess the variability and information content of these spike patterns. By quantifying the uncertainty and informational content of neuronal patterns, entropy measures provide insights into neural coding strategies, synaptic plasticity, network dynamics, and cognitive processes. Here, we review basic entropy metrics and then we provide examples of recent advancements in using entropy as a tool to improve our understanding of neuronal processing. It focuses especially on studies on critical dynamics in neural networks and the relation of entropy to predictive coding and cortical communication. We highlight the necessity of expanding entropy measures from single neurons to encompass multi-neuronal activity patterns, as cortical circuits communicate through coordinated spatiotemporal activity patterns, called neuronal packets. We discuss how the sequential and partially stereotypical nature of neuronal packets influences the entropy of cortical communication. Stereotypy reduces entropy by enhancing reliability and predictability in neural signaling, while variability within packets increases entropy, allowing for greater information capacity. This balance between stereotypy and variability supports both robustness and flexibility in cortical information processing. We also review challenges in applying entropy to analyze such spatiotemporal neuronal spike patterns, notably, the “curse of dimensionality” in estimating entropy for high-dimensional neuronal data. Finally, we discuss strategies to overcome these challenges, including dimensionality reduction techniques, advanced entropy estimators, sparse coding schemes, and the integration of machine learning approaches. Thus, this work summarizes the most recent developments on how entropy measures contribute to our understanding of principles underlying neural coding.

Deep-learning models reveal how context and listener attention shape electrophysiological correlates of speech-to-language transformation.

To transform continuous speech into words, the human brain must resolve variability across utterances in intonation, speech rate, volume, accents and so on. A promising approach to explaining this process has been to model electroencephalogram (EEG) recordings of brain responses to speech. Contemporary models typically invoke context invariant speech categories (e.g. phonemes) as an intermediary representational stage between sounds and words. However, such models may not capture the complete picture because they do not model the brain mechanism that categorizes sounds and consequently may overlook associated neural representations. By providing end-to-end accounts of speech-to-text transformation, new deep-learning systems could enable more complete brain models. We model EEG recordings of audiobook comprehension with the deep-learning speech recognition system Whisper. We find that (1) Whisper provides a self-contained EEG model of an intermediary representational stage that reflects elements of prelexical and lexical representation and prediction; (2) EEG modeling is more accurate when informed by 5-10s of speech context, which traditional context invariant categorical models do not encode; (3) Deep Whisper layers encoding linguistic structure were more accurate EEG models of selectively attended speech in two-speaker "cocktail party" listening conditions than early layers encoding acoustics. No such layer depth advantage was observed for unattended speech, consistent with a more superficial level of linguistic processing in the brain.

DGNMDA: Dual Heterogeneous Graph Neural Network Encoder for miRNA-Disease Association Prediction

In recent years, numerous studies have highlighted the pivotal importance of miRNAs in personalized healthcare, showcasing broad application prospects. miRNAs hold significant potential in disease diagnosis, prognosis assessment, and therapeutic target discovery, making them an integral part of precision medicine. They are expected to enable precise disease subtyping and risk prediction, thereby advancing the development of precision medicine. GNNs, a class of deep learning architectures tailored for graph data analysis, have greatly facilitated the advancement of miRNA-disease association prediction algorithms. However, current methods often fall short in leveraging network node information, particularly in utilizing global information while neglecting the importance of local information. Effectively harnessing both local and global information remains a pressing challenge. To tackle this challenge, we propose an innovative model named DGNMDA. Initially, we constructed various miRNA and disease similarity networks based on authoritative databases. Subsequently, we creatively design a dual heterogeneous graph neural network encoder capable of efficiently learning feature information between adjacent nodes and similarity information across the entire graph. Additionally, we develop a specialized fine-grained multi-layer feature interaction gating mechanism to integrate outputs from the neural network encoders to identify novel associations connecting miRNAs with diseases. We evaluate our model using 5-fold cross-validation and real-world disease case studies, based on the HMDD V3.2 dataset. Our method demonstrates superior performance compared to existing approaches in various tasks, confirming the effectiveness and potential of DGNMDA as a robust method for predicting miRNA-disease associations.

Activity in Occipito-Temporal Cortex Is Involved in Tool-Use Planning and Contributes to Tool-Related Semantic Neural Representations

Abstract Tool use and language are highly refined human abilities which may show neural commonalities due to their potential reciprocal interaction during evolution. Recent work provided evidence for shared neural resources between tool use and syntax. However, whether activity within the tool-use network also contributes to semantic neural representations of tool nouns remains untested. To this aim, we identified the tool-use planning network with functional magnetic resonance imaging while participants used pliers. The very same participants underwent a semantic priming task including two categories, tool nouns and animal nouns, to highlight the respective underlying networks. With multivariate analyses of the activation neural patterns, we tested whether activity in tool-use brain clusters takes part in the neural representation of tool nouns as compared with animal nouns. The results revealed that word semantic categories were decoded within the left occipito-temporal cortex activated by preparing to use a tool, with similar patterns of brain activity for words within the same category. In addition, in the same area, neural activations for tool nouns were found to be higher than those for animal nouns. These findings suggest that activity in tool-use related brain areas encodes semantic information separately for tool nouns and animal nouns, thus supporting the embodiment of tool-noun processing in the tool-use sensorimotor network.

Integrating Vision: From Neuroscience to Artificial Intelligence

Abstract. This review article explores the complex interplay between neuroscience and artificial intelligence, focusing on vision processing and its applications. It starts by outlining the biological basis of vision, delving into how visual information is processed and encoded in the brain. The discussion then transitions to artificial intelligence, particularly machine vision, highlighting the advancements and technologies that mimic biological processes. A critical analysis compares how visual information is utilized in both biological organisms and artificial systems, with an emphasis on cognitive functions and neural encoding. The challenges of integrating vision across various sensory modalities are examined, underscoring the technological and cognitive limitations currently faced. The review culminates by identifying potential research paths aimed at closing the gap between neuroscience and AI. This involves enhancing the understanding and functionality of vision in multisensory contexts, striving to foster a more comprehensive approach to artificial intelligence that mirrors the complexity of human perception.

Deep learning-based real-time ray tracing technology in games

Abstract. In recent years, deep learning-based techniques have revolutionized real-time ray tracing for gaming, significantly enhancing visual fidelity and rendering performance. This paper reviews various state-of-the-art methods, including the use of Generative Adversarial Networks (GANs) for realistic shading, the use of neural temporal adaptive sampling, the use of subpixel sampling reconstruction, and the use of neural scene representation. Key findings highlight improvements in perceived realism, temporal stability, image fidelity, and computational efficiency. Techniques such as neural intersection functions and spatiotemporal reservoir resampling further optimize rendering speed and memory usage. Additionally, approaches like adaptive sampling and neural denoising using layer embeddings contribute to reduced noise and enhanced image clarity. Collectively, these advancements make real-time ray tracing more feasible for high-fidelity gaming applications, offering enhanced graphics without compromising performance. These improvements are significant. My analysis underscores the critical role of deep learning in overcoming traditional ray tracing challenges, paving the way for more immersive and responsive gaming experiences. Furthermore, these innovations suggest a promising future for integrating advanced ray tracing techniques into a broader range of interactive media, ensuring both visual excellence and operational efficiency.

Distinct inhibitory neurons differently shape neuronal codes for sound intensity in the auditory cortex.

Cortical circuits contain multiple types of inhibitory neurons which shape how information is processed within neuronal networks. Here, we asked whether somatostatin-expressing (SST) and vasoactive intestinal peptide-expressing (VIP) inhibitory neurons have distinct effects on population neuronal responses to noise bursts of varying intensities. We optogenetically stimulated SST or VIP neurons while simultaneously measuring the calcium responses of populations of hundreds of neurons in the auditory cortex of male and female awake, head-fixed mice to sounds. Upon SST neuronal activation, noise bursts representations became more discrete for different intensity levels, relying on cell identity rather than strength. By contrast, upon VIP neuronal activation, noise bursts of different intensity level activated overlapping neuronal populations, albeit at different response strengths. At the single-cell level, SST and VIP neuronal activation differentially modulated the response-level curves of monotonic and nonmonotonic neurons. SST neuronal activation effects were consistent with a shift of the neuronal population responses toward a more localist code with different cells responding to sounds of different intensity. By contrast, VIP neuronal activation shifted responses towards a more distributed code, in which sounds of different intensity level are encoded in the relative response of similar populations of cells. These results delineate how distinct inhibitory neurons in the auditory cortex dynamically control cortical population codes. Different inhibitory neuronal populations may be recruited under different behavioral demands, depending on whether categorical or invariant representations are advantageous for the task.Significance Statement Information about sounds is represented in the auditory cortex by neuronal population activity that has a characteristic sparse structure. Cortical neuronal populations comprise multiple types of excitatory and inhibitory neurons. Here, we find that activating different types of inhibitory neurons differentially controls population neuronal representations, with one type of inhibitory neurons increasing the differences in the identity of the cells recruited to represent the different sounds, and another inhibitory neuron type changing the relative activity level of overlapping neuronal populations. Such transformations may be beneficial for different types of auditory behaviors, suggesting that these different types of inhibitory neurons may be recruited under different behavioral constraints in optimizing neuronal representations of sounds.

Multitasking practice eliminates modality-based interference by separating task representations in sensory brain regions.

The debate on the neural basis of multitasking costs evolves around neural overlap between concurrently performed tasks. Recent evidence suggests that training-related reductions in representational overlap in fronto-parietal brain regions predict multitasking improvements. Cognitive theories assume that overlap of task representations may lead to unintended information exchange between tasks (i.e., crosstalk). Modality-based crosstalk was suggested as a source for multitasking costs in multisensory settings. Robust findings of increased costs for certain modality mappings may be explained by crosstalk between the stimulus modality in one task and sensory action consequences in the concurrently performed task. Whether modality-based crosstalk emerges from representational overlap in general fronto-parietal multitasking regions or modality-specific regions is not known yet. In this functional neuroimaging study, we investigate neural overlap during multitasking performance in humans, focusing on modality compatibility by employing multivariate pattern analysis and modality-specific practice interventions in three groups (total N = 54, 24 females). We observed significant differences between modality compatible and modality incompatible single-task representations, specifically in the auditory cortex but not in fronto-parietal regions. Notably, improved auditory decoding accuracy related to modality incompatible tasks was predictive of performance gains in the corresponding dual task along with complete elimination of modality-specific dual-task costs. This predictive relationship was evident only in the group practicing modality incompatible mappings, suggesting that specific practice on task sets with modality overlap influenced both neural representations and subsequent multitasking performance. This study contributes to the integration of cognitive theory and neuroscience and the role of task representations in dual-task interference.Significance Statement In a society dominated by multitasking, understanding its neurocognitive basis and plasticity is crucial for key aspects of everyday tasks. We investigate the neural mechanisms behind multitasking limitations, offering insights for targeted cognitive interventions. The study builds upon established theories of cognitive multitasking and imaging research, addressing the concept of modality-based crosstalk - the unintended exchange of modality-based information between tasks. Through functional brain imaging and pattern analysis, we examined how neural task representations contribute to performance costs in dual tasks with varying degrees of modality overlap. Notably, our findings demonstrate a practice-related decrease in neural overlap which is associated with substantial multitasking improvements, specifically in the auditory cortex, emphasizing the contribution of sensory regions to flexible multidimensional task representations.

Conflict during learning reconfigures the neural representation of positive valence and approach behavior.

Punishing and rewarding experiences can change the valence of sensory stimuli and guide animal behavior in opposite directions, resulting in avoidance or approach. Often, however, a stimulus is encountered with both positive and negative experiences. How is such conflicting information represented in the brain and resolved into a behavioral decision? We address this question by dissecting a circuit for sexual conditioning in C.elegans. In this learning paradigm, an odor is conditioned with both a punishment (starvation) and a reward (mates), resulting in odor approach. We find that negative and positive experiences are both encoded by the neuropeptide pigment dispersing factor 1 (PDF-1) being released from, and acting on, different neurons. Each experience creates a distinct memory in the circuit for odor processing. This results in the sensorimotor representation of the odor being different in naive and sexually conditioned animals, despite both displaying approach. Our results reveal that the positive valence of a stimulus is not represented in the activity of any single neuron class but flexibly represented within the circuit according to the experiences and predictions associated with the stimulus.

A 7T fMRI investigation of hand and tool areas in the lateral and ventral occipitotemporal cortex.

Previous studies demonstrated the existence of hand and tool areas in lateral and ventral occipitotemporal cortex (OTC), as well as an overlap between them. We reinvestigated this organization using 7T fMRI, benefiting from a higher signal-to-noise ratio than 3T. This enabled us to include a wider array of categories to achieve a more holistic perspective, and to omit certain spatial preprocessing steps. Despite these improvements, univariate analysis confirmed the existence of hand-tool overlap across OTC, which is striking given the omission of the spatial preprocessing steps that can influence overlap. There was significantly more overlap between hands and tools, compared to other overlap types in the left hemisphere of OTC. The overlap was also larger in the left lateral OTC as compared to the right lateral OTC. We found in all hand areas a differentiation between tools and other types of objects, although they still responded more to bodies than to tools. Regarding the tool areas, we observed a differentiation between hands and other categories such as faces and non-tool objects. Left hemisphere tool areas also differentiated between hands and bodies. When excluding the overlapping voxels from the hand and tool areas, they still showed a significant response to tools or hands (compared to objects or faces) respectively. Multi-voxel pattern analysis indicated that neural representations in the hand areas showed greater similarity between hands and tools than between hands and other objects. In the tool areas, the neural representations between tools and hands and between tools and other type of objects were all equally similar. To summarize, capitalizing on the benefits of 7T fMRI, we further substantiate the evidence in favor of hand-tool overlap in several regions of occipitotemporal cortex.

Neural representations of phonological information in bilingual language production.

Previous research has explored the neural mechanisms of bilinguals' language production, but most studies focused on neural mechanisms of cognitive control during language production. Therefore, it is unclear which brain regions represent lexical information (especially phonological information) during production and how they are affected by language context. To address those questions, we used representational similarity analysis to explore neural representations of phonological information in native (L1) and second languages (L2) in the single- and mixed-language contexts, respectively. Results showed that Chinese-English bilinguals behaviorally performed worse and exhibited more activations in brain regions associated with language processing and cognitive control in the mixed-language context relative to the single-language context. Further representational similarity analysis revealed that phonological representations of L1 were detected in the left pars opercularis, middle frontal gyrus, and anterior supramarginal gyrus, while phonological representations of L2 were detected in the bilateral occipitotemporal cortex regardless of the target language. More interestingly, robust phonological representations of L1 were observed in brain areas related to phonological processing during L2 production regardless of language context. These results provide direct neuroimaging evidence for the nonselective processing hypothesis and highlight the superiority of phonological representations in the dominant language during bilingual language production.

Lower Limb Motion Recognition Based on sEMG and CNN-TL Fusion Model.

To enhance the classification accuracy of lower limb movements, a fusion recognition model integrating a surface electromyography (sEMG)-based convolutional neural network, transformer encoder, and long short-term memory network (CNN-Transformer-LSTM, CNN-TL) was proposed in this study. By combining these advanced techniques, significant improvements in movement classification were achieved. Firstly, sEMG data were collected from 20 subjects as they performed four distinct gait movements: walking upstairs, walking downstairs, walking on a level surface, and squatting. Subsequently, the gathered sEMG data underwent preprocessing, with features extracted from both the time domain and frequency domain. These features were then used as inputs for the machine learning recognition model. Finally, based on the preprocessed sEMG data, the CNN-TL lower limb action recognition model was constructed. The performance of CNN-TL was then compared with that of the CNN, LSTM, and SVM models. The results demonstrated that the accuracy of the CNN-TL model in lower limb action recognition was 3.76%, 5.92%, and 14.92% higher than that of the CNN-LSTM, CNN, and SVM models, respectively, thereby proving its superior classification performance. An effective scheme for improving lower limb motor function in rehabilitation and assistance devices was thus provided.

The generation of a neuronal phase code for space.

The generation of a neuronal phase code for space.

Differential neural representations of syntactic and semantic information across languages in Chinese-English bilinguals

Differential neural representations of syntactic and semantic information across languages in Chinese-English bilinguals

Registered Report Stage II: Decoding the category information from evoked potentials to visible and invisible visual objects

Previous studies that use decoding methods and EEG to investigate the neural representation of the category information of visual objects focused mainly on consciously processed visual objects. It remains unclear whether the category information of unconsciously processed visual objects can be decoded and whether the decoding performance is different for consciously and unconsciously processed visual objects. The present study compared the neural decoding of the animacy category of visible and invisible visual objects via EEG and decoding methods. The results revealed that the animacy of visible visual objects could be decoded above the chance level by the P200, N300, and N400, but not by the early N/P100. However, the animacy of invisible visual objects could not be decoded above the chance level by neither early nor late ERP components. The decoding accuracy was greater for visible visual objects than that for invisible visual objects for the P200, N300 and N400. These results suggested that access to animacy category information for visual objects requires conscious processing.