Neural Representation Research Articles

Hippocampal representations of alternative possibilities are flexibly generated to meet cognitive demands.

The cognitive ability to go beyond the present to consider alternative possibilities, including potential futures and counterfactual pasts, can support adaptive decision making. Complex and changing real-world environments, however, have many possible alternatives. Whether and how the brain can select among them to represent alternatives that meet current cognitive needs remains unknown. We therefore examined neural representations of alternative spatial locations in the rat hippocampus during navigation in a complex patch foraging environment with changing reward probabilities. We found representations of multiple alternatives along paths ahead and behind the animal, including in distant alternative patches. Critically, these representations were modulated in distinct patterns across successive trials: alternative paths were represented proportionate to their evolving relative value and predicted subsequent decisions, whereas distant alternatives were prevalent during value updating. These results demonstrate that the brain modulates the generation of alternative possibilities in patterns that meet changing cognitive needs for adaptive behavior.

Distinct Neural Representations of Hunger and Thirst in Neonatal Mice before the Emergence of Food- and Water-seeking Behaviors.

Hunger and thirst are two fundamental drives for maintaining homeostasis, and elicit distinct food- and water-seeking behaviors essential for survival. For neonatal mammals, however, both hunger and thirst are sated by consuming milk from their mother. While distinct neural circuits underlying hunger and thirst drives in the adult brain have been characterized, it is unclear when these distinctions emerge in neonates and what processes may affect their development. Here we show that hypothalamic hunger and thirst regions already exhibit specific responses to starvation and dehydration well before a neonatal mouse can seek food and water separately. At this early age, hunger drives feeding behaviors more than does thirst. Within neonatal regions that respond to both hunger and thirst, subpopulations of neurons respond distinctly to one or the other need. Combining food and water into a liquid diet throughout the animal's life does not alter the distinct representations of hunger and thirst in the adult brain. Thus, neural representations of hunger and thirst become distinct before food- and water-seeking behaviors mature and are robust to environmental changes in food and water sources.

Rational inattention in neural coding for economic choice.

Mental operations like computing the value of an option are computationally expensive. Even before we evaluate options, we must decide how much attentional effort to invest in the evaluation process. More precise evaluation will improve choice accuracy, and thus reward yield, but the gain may not justify the cost. Rational Inattention theories provide an accounting of the internal economics of attentionally effortful economic decisions. To understand this process, we examined choices and neural activity in several brain regions in six macaques making risky choices. We extended the rational inattention framework to incorporate the foraging theoretic understanding of local environmental richness or reward rate, which we predict will promote attentional effort. Consistent with this idea, we found local reward rate positively predicted choice accuracy. Supporting the hypothesis that this effect reflects variations in attentional effort, richer contexts were associated with increased baseline and evoked pupil size. Neural populations likewise showed systematic baseline coding of reward rate context. During increased reward rate contexts, ventral striatum and orbitofrontal cortex showed both an increase in value decodability and a rotation in the population geometries for value. This confluence of these results suggests a mechanism of attentional effort that operates by controlling gain through using partially distinct population codes for value. Additionally, increased reward rate accelerated value code dynamics, which have been linked to improved signal-to-noise. These results extend the theory of rational inattention to static and stationary contexts and align theories of rational inattention with specific costly, neural processes.

A neural mechanism for learning from delayed postingestive feedback.

Animals learn the value of foods based on their postingestive effects and thereby develop aversions to foods that are toxic1-6 and preferences to those that are nutritious7-14. However, it remains unclear how the brain is able to assign credit to flavors experienced during a meal with postingestive feedback signals that can arise after a substantial delay. Here, we reveal an unexpected role for postingestive reactivation of neural flavor representations in this temporal credit assignment process. To begin, we leverage the fact that mice learn to associate novel15-18, but not familiar, flavors with delayed gastric malaise signals to investigate how the brain represents flavors that support aversive postingestive learning. Surveying cellular resolution brainwide activation patterns reveals that a network of amygdala regions is unique in being preferentially activated by novel flavors across every stage of the learning process: the initial meal, delayed malaise, and memory retrieval. By combining high-density recordings in the amygdala with optogenetic stimulation of genetically defined hindbrain malaise cells, we find that postingestive malaise signals potently and specifically reactivate amygdalar novel flavor representations from a recent meal. The degree of malaise-driven reactivation of individual neurons predicts strengthening of flavor responses upon memory retrieval, leading to stabilization of the population-level representation of the recently consumed flavor. In contrast, meals without postingestive consequences degrade neural flavor representations as flavors become familiar and safe. Thus, our findings demonstrate that interoceptive reactivation of amygdalar flavor representations provides a neural mechanism to resolve the temporal credit assignment problem inherent to postingestive learning.

A geometrical solution underlies general neural principle for serial ordering

A general mathematical description of how the brain sequentially encodes knowledge remains elusive. We propose a linear solution for serial learning tasks, based on the concept of mixed selectivity in high-dimensional neural state spaces. In our framework, neural representations of items in a sequence are projected along a “geometric” mental line learned through classical conditioning. The model successfully solves serial position tasks and explains behaviors observed in humans and animals during transitive inference tasks amidst noisy sensory input and stochastic neural activity. This approach extends to recurrent neural networks performing motor decision tasks, where the same geometric mental line correlates with motor plans and modulates network activity according to the symbolic distance between items. Serial ordering is thus predicted to emerge as a monotonic mapping between sensory input and behavioral output, highlighting a possible pivotal role for motor-related associative cortices in transitive inference tasks.

The role of binocular disparity and attention in the neural representation of multiple moving stimuli in the visual cortex.

Segmenting visual scenes into distinct objects and surfaces is a fundamental visual process, with stereoscopic depth and motion serving as crucial cues. However, how the visual system uses these cues to segment multiple objects is not fully understood. We investigated how neurons in the middle-temporal (MT) cortex of macaque monkeys represent overlapping surfaces at different depths, moving in different directions. Neuronal activity was recorded from three male monkeys during discrimination tasks under varying attention conditions. We found that neuronal responses to overlapping surfaces showed a robust bias toward the binocular disparity of one surface over the other. The disparity bias of a neuron was positively correlated with the neuron's disparity preference for a single surface. In two animals, neurons preferring near disparities of single surfaces (near neurons) showed a near bias for overlapping stimuli, while neurons preferring far disparities (far neurons) showed a far bias. In the third animal, both near and far neurons displayed a near bias, though the near neurons showed a stronger near bias. All three animals exhibited an initial near bias across neurons relative to the average of the responses to the individual surfaces. Although attention modulated neuronal responses, the disparity bias was not caused by attention. We also found that the effect of attention was consistent with object-based, rather than feature-based attention. We proposed a model in which the pool size of the neuron population that weighs the responses to individual stimulus components can be variable. This model is a novel extension of the standard normalization model and provides a unified explanation for the disparity bias across animals. Our results reveal how MT neurons encode multiple stimuli moving at different depths and present new evidence of response modulation by object-based attention. The disparity bias allows subgroups of neurons to preferentially represent individual surfaces of multiple stimuli at different depths, thereby facilitating segmentation.

Space-time reconstruction for lensless imaging using implicit neural representations

Many computational imaging inverse problems are challenged by noise, model mismatch, and other imperfections that decrease reconstruction quality. For data taken sequentially in time, instead of reconstructing each frame independently, space-time algorithms simultaneously reconstruct multiple frames, thereby taking advantage of temporal redundancy through space-time priors. This helps with denoising and provides improved reconstruction quality, but often requires significant computational and memory resources. Designing effective but flexible temporal priors is also challenging. Here, we propose using an implicit neural representation to model dynamics and act as a computationally tractable and flexible space-time prior. We demonstrate this approach on video captured with a lensless imager, DiffuserCam, and show improved reconstruction results and robustness to noise compared to frame-by-frame methods.

Feature radiance fields (FeRF): A multi-level feature fusion method with deep neural network for image synthesis

Neural Radiance Field (NeRF) has brought revolutionary changes to the field of image synthesis with its unique ability to generate highly realistic multi-view consistent images from a neural scene representation. However, current NeRF-based methods still largely depend on multiple, precisely posed images, especially for complex or dynamic scenes, limiting their versatility. Furthermore, some recent strategies attempt to integrate simple feature extraction networks with volume rendering techniques to reduce multi-view dependence but create blurry outputs, highlighting the need for more sophisticated feature handling to unlock NeRF's full potential. In this paper, we propose an image synthesis method named FeRF, distinguished by its capacity to perform comprehensive feature extraction on individual unposed images and facilitate feature fusion at any stage. Additionally, we present an "elaborated-feature generation network" (EGN) composed of four modules, which is configured with two advanced feature extraction modules aimed at precisely refining and processing subtle, complex visual features from a single image. Given that the core objective of FeRF is the precise capture and processing of intricate features from the input images, we innovatively incorporated precisely designed attention mechanisms into the network architecture to optimize and highlight the importance of key feature attributes, thereby effectively enhancing their contribution to subsequent volume rendering processes. Extensive experimentation validates the outstanding qualitative and quantitative performance of our proposed network structure. In comparison to current image feature-based generalized image synthesis methods, it achieves superior reconstruction quality and level of detail.

Neural Activity Differentiates Novel and Learned Event Boundaries.

People parse continuous experiences at natural breakpoints called event boundaries, which is important for understanding an environment's causal structure and for responding to uncertainty within it. However, it remains unclear how different forms of uncertainty affect the parsing of continuous experiences and how such uncertainty influences the brain's processing of ongoing events. We exposed human participants of both sexes (N = 34) to a continuous sequence of semantically meaningless images. We generated sequences from random walks through a graph that grouped images into temporal communities. After learning, we asked participants to segment another sequence at natural breakpoints (event boundaries). Participants segmented the sequence at learned transitions between communities, as well as at novel transitions, suggesting that people can segment temporally extended experiences into events based on learned structure as well as prediction error. Greater segmentation at novel boundaries was associated with enhanced parietal scalp electroencephalography (EEG) activity between 250 and 450 ms after the stimulus onset. Multivariate classification of EEG activity showed that novel and learned boundaries evoked distinct patterns of neural activity, particularly theta band power in posterior electrodes. Learning also led to distinct neural representations for stimuli within the temporal communities, while neural activity at learned boundary nodes showed predictive evidence for the adjacent community. The data show that people segment experiences at both learned and novel boundaries and suggest that learned event boundaries trigger retrieval of information about the upcoming community that could underlie anticipation of the next event in a sequence.

Context-dependent neural preparation for information relevance vs. probability

Abstract Preparation is a top-down phenomenon known to improve performance across different situations. In light of recent electrophysiological findings that suggest that anticipatory neural preactivations linked to preparation are context-specific and do not generalize across domains, in the current study we used fMRI to investigate the brain regions involved in these differential patterns. We applied multivariate decoding to data obtained in a paradigm where, in different blocks, cues provided information about the relevance or probability of incoming target stimuli. Results showed that the anticipated stimulus category was pre-activated in both conditions, mostly in different brain regions within the ventral visual cortex and with differential overlap with actual target perception. Crucially, there was scarce cross-classification across attention and expectation contexts except on a patch of the fusiform gyrus, indicating mostly differential neural coding of anticipated contents in relevance and probability scenarios. Finally, a model-based fMRI-EEG fusion showed that these regions differentially code for specific conditions during preparation, as well as specifically preparing for category anticipation in a ramping-up manner. Overall, our results stress the specificity of anticipatory neural processing depending on its informative role while highlighting a key hub of commonality in the fusiform gyrus.

Generalized Encoding of the Relative Subjective Value of Cognitive Effort in the Dorsal ACC.

Making choices about whether and when to engage cognitive effort are a common feature of everyday experience, with important consequences for academic, career, and health outcomes. Yet, despite their hypothesized importance, very little is understood about the underlying mechanisms that support this form of human cost-benefit decision-making. To investigate these mechanisms, we used the Cognitive Effort Discounting Paradigm (Cog-ED) during fMRI scanning to precisely quantify the neural encoding of varying cognitive effort demands relative to reward outcomes, within two distinct cognitive domains (working memory, speech comprehension). The findings provide strong evidence that the dorsal anterior cingulate cortex (dACC) plays a central and selective role in this decision-making process. Trial-by-trial modulations in dACC activation tracked the relative subjective value of the low-effort, low-reward option, with the strongest activity occurring when this was of greater value than the high-effort, high-reward option. In contrast, dACC activity was not modulated by decision difficulty, though such effects were found in other frontoparietal regions. Critically, dACC activity was also strongly correlated across the two decision-making task domains and further predicted subsequent choice behavior in both. Together, the results suggest that dACC activity modulation reflects a domain-general valuation comparison mechanism, which acts to bias participants away from decisions to engage in cognitive effort, when the perceived subjective costs of such engagement outweigh the reward-related benefits. These findings complement work in other cost domains and species by pointing to a clear role of the dACC in representing subjective value differences between choice options during cost-benefit decision-making.

The tilt illusion arises from an efficient reallocation of neural coding resources at the contextual boundary.

The tilt illusion - a bias in the perceived orientation of a center stimulus induced by an oriented surround - illustrates how context shapes visual perception. While the tilt illusion has been the subject of quantitative study for over 85 years, we still lack a comprehensive account of the phenomenon that connects its neural and behavioral characteristics. Here, we demonstrate that the tilt illusion originates from a dynamic change in neural coding precision induced by the surround context. We simultaneously obtained psychophysical and fMRI responses from human subjects while they viewed gratings in the absence and presence of an oriented surround, and extracted sensory encoding precision from their behavioral and neural data. Both measures show that in the absence of a surround, encoding reflects the natural scene statistics of orientation. However, in the presence of an oriented surround, encoding precision is significantly increased for stimuli similar to the surround orientation. This local change in encoding is sufficient to accurately predict the behavioral characteristics of the tilt illusion using a Bayesian observer model. The effect of surround modulation increases along the ventral stream, and is localized to the portion of the visual cortex with receptive fields at the center-surround boundary. The pattern of change in coding accuracy reflects the surround-conditioned orientation statistics in natural scenes, but cannot be explained by local stimulus configuration. Our results suggest that the tilt illusion naturally emerges from a dynamic coding strategy that efficiently reallocates neural coding resources based on the current stimulus context.

A novel DRL-guided sparse voxel decoding model for reconstructing perceived images from brain activity

BackgroundDue to the sparse encoding character of the human visual cortex and the scarcity of paired training samples for {images, fMRIs}, voxel selection is an effective means of reconstructing perceived images from fMRI. However, the existing data-driven voxel selection methods have not achieved satisfactory results. New methodHere, a novel deep reinforcement learning-guided sparse voxel (DRL-SV) decoding model is proposed to reconstruct perceived images from fMRI. We innovatively describe voxel selection as a Markov decision process (MDP), training agents to select voxels that are highly involved in specific visual encoding. ResultsExperimental results on two public datasets verify the effectiveness of the proposed DRL-SV, which can accurately select voxels highly involved in neural encoding, thereby improving the quality of visual image reconstruction. Comparison with existing methodsWe qualitatively and quantitatively compared our results with the state-of-the-art (SOTA) methods, getting better reconstruction results. We compared the proposed DRL-SV with traditional data-driven baseline methods, obtaining sparser voxel selection results, but better reconstruction performance. ConclusionsDRL-SV can accurately select voxels involved in visual encoding on few-shot, compared to data-driven voxel selection methods. The proposed decoding model provides a new avenue to improving the image reconstruction quality of the primary visual cortex.

Neurophysiological evidence of sensory prediction errors driving speech sensorimotor adaptation.

The human sensorimotor system has a remarkable ability to quickly and efficiently learn movements from sensory experience. A prominent example is sensorimotor adaptation, learning that characterizes the sensorimotor system's response to persistent sensory errors by adjusting future movements to compensate for those errors. Despite being essential for maintaining and fine-tuning motor control, mechanisms underlying sensorimotor adaptation remain unclear. A component of sensorimotor adaptation is implicit (i.e., the learner is unaware of the learning process) which has been suggested to result from sensory prediction errors-the discrepancies between predicted sensory consequences of motor commands and actual sensory feedback. However, to date no direct neurophysiological evidence that sensory prediction errors drive adaptation has been demonstrated. Here, we examined prediction errors via magnetoencephalography (MEG) imaging of the auditory cortex (n = 34) during sensorimotor adaptation of speech to altered auditory feedback, an entirely implicit adaptation task. Specifically, we measured how speaking-induced suppression (SIS)--a neural representation of auditory prediction errors--changed over the trials of the adaptation experiment. SIS refers to the suppression of auditory cortical response to speech onset (in particular, the M100 response) to self-produced speech when compared to the response to passive listening to identical playback of that speech. SIS was reduced (reflecting larger prediction errors) during the early learning phase compared to the initial unaltered feedback phase. Furthermore, reduction in SIS positively correlated with behavioral adaptation extents, suggesting that larger prediction errors were associated with more learning. In contrast, such a reduction in SIS was not found in a control experiment in which participants heard unaltered feedback and thus did not adapt. In addition, in some participants who reached a plateau in the late learning phase, SIS increased (reflecting smaller prediction errors), demonstrating that prediction errors were minimal when there was no further adaptation. Together, these findings provide the first neurophysiological evidence for the hypothesis that prediction errors drive human sensorimotor adaptation.

Three-Dimensional Reconstruction of Indoor Scenes Based on Implicit Neural Representation.

Reconstructing 3D indoor scenes from 2D images has always been an important task in computer vision and graphics applications. For indoor scenes, traditional 3D reconstruction methods have problems such as missing surface details, poor reconstruction of large plane textures and uneven illumination areas, and many wrongly reconstructed floating debris noises in the reconstructed models. This paper proposes a 3D reconstruction method for indoor scenes that combines neural radiation field (NeRFs) and signed distance function (SDF) implicit expressions. The volume density of the NeRF is used to provide geometric information for the SDF field, and the learning of geometric shapes and surfaces is strengthened by adding an adaptive normal prior optimization learning process. It not only preserves the high-quality geometric information of the NeRF, but also uses the SDF to generate an explicit mesh with a smooth surface, significantly improving the reconstruction quality of large plane textures and uneven illumination areas in indoor scenes. At the same time, a new regularization term is designed to constrain the weight distribution, making it an ideal unimodal compact distribution, thereby alleviating the problem of uneven density distribution and achieving the effect of floating debris removal in the final model. Experiments show that the 3D reconstruction effect of this paper on ScanNet, Hypersim, and Replica datasets outperforms the state-of-the-art methods.

Models optimized for real-world tasks reveal the task-dependent necessity of precise temporal coding in hearing.

Neurons encode information in the timing of their spikes in addition to their firing rates. Spike timing is particularly precise in the auditory nerve, where action potentials phase lock to sound with sub-millisecond precision, but its behavioral relevance remains uncertain. We optimized machine learning models to perform real-world hearing tasks with simulated cochlear input, assessing the precision of auditory nerve spike timing needed to reproduce human behavior. Models with high-fidelity phase locking exhibited more human-like sound localization and speech perception than models without, consistent with an essential role in human hearing. However, the temporal precision needed to reproduce human-like behavior varied across tasks, as did the precision that benefited real-world task performance. These effects suggest that perceptual domains incorporate phase locking to different extents depending on the demands of real-world hearing. The results illustrate how optimizing models for realistic tasks can clarify the role of candidate neural codes in perception.

Neural Representations that Reveal a Unified Continuum between Physical and Social Events

Neural Representations that Reveal a Unified Continuum between Physical and Social Events

How Shape Perception Works, in Two Dimensions and Three Dimensions

The ventral visual pathway transforms retinal images into neural representations that support object understanding, including exquisite appreciation of precise 2D pattern shape and 3D volumetric shape. We articulate a framework for understanding the goals of this transformation and how they are achieved by neural coding at successive ventral pathway stages. The critical goals are (a) radical compression to make shape information communicable across axonal bundles and storable in memory, (b) explicit coding to make shape information easily readable by the rest of the brain and thus accessible for cognition and behavioral control, and (c) representational stability to maintain consistent perception across highly variable viewing conditions. We describe how each transformational step in ventral pathway vision serves one or more of these goals. This three-goal framework unifies discoveries about ventral shape processing into a neural explanation for our remarkable experience of shape as a vivid, richly detailed aspect of the natural world.

Visual motion extrapolation of moving objects drives real-time temporal re-alignment across hierarchical neural position representations

Visual motion extrapolation of moving objects drives real-time temporal re-alignment across hierarchical neural position representations

From point light displays to rich social narratives: neural representations of visual social processing in the superior temporal sulcus

From point light displays to rich social narratives: neural representations of visual social processing in the superior temporal sulcus