Anterior Inferotemporal Cortex Research Articles

Bodies in motion: Unraveling the distinct roles of motion and shape in dynamic body responses in the temporal cortex

The temporal cortex represents social stimuli, including bodies. We examine and compare the contributions of dynamic and static features to the single-unit responses to moving monkey bodies in and between a patch in the anterior dorsal bank of the superior temporal sulcus (dorsal patch [DP]) and patches in the anterior inferotemporal cortex (ventral patch [VP]), using fMRI guidance in macaques. The response to dynamics varies within both regions, being higher in DP. The dynamic body selectivity of VP neurons correlates with static features derived from convolutional neural networks and motion. DP neurons' dynamic body selectivity is not predicted by static features but is dominated by motion. Whereas these data support the dominance of motion in the newly proposed "dynamic social perception" stream, they challenge the traditional view that distinguishes DP and VP processing in terms of motion versus static features, underscoring the role of inferotemporal neurons in representing body dynamics.

Object representation in a gravitational reference frame.

When your head tilts laterally, as in sports, reaching, and resting, your eyes counterrotate less than 20%, and thus eye images rotate, over a total range of about 180°. Yet, the world appears stable and vision remains normal. We discovered a neural strategy for rotational stability in anterior inferotemporal cortex (IT), the final stage of object vision in primates. We measured object orientation tuning of IT neurons in macaque monkeys tilted +25 and -25° laterally, producing ~40° difference in retinal image orientation. Among IT neurons with consistent object orientation tuning, 63% remained stable with respect to gravity across tilts. Gravitational tuning depended on vestibular/somatosensory but also visual cues, consistent with previous evidence that IT processes scene cues for gravity's orientation. In addition to stability across image rotations, an internal gravitational reference frame is important for physical understanding of a world where object position, posture, structure, shape, movement, and behavior interact critically with gravity.

Dissociation and hierarchy of human visual pathways for simultaneously coding facial identity and expression

Humans have an extraordinary ability to recognize facial expression and identity from a single face simultaneously and effortlessly, however, the underlying neural computation is not well understood. Here, we optimized a multi-task deep neural network to classify facial expression and identity simultaneously. Under various optimization training strategies, the best-performing model consistently showed ‘share-separate’ organization. The two separate branches of the best-performing model also exhibited distinct abilities to categorize facial expression and identity, and these abilities increased along the facial expression or identity branches toward high layers. By comparing the representational similarities between the best-performing model and functional magnetic resonance imaging (fMRI) responses in the human visual cortex to the same face stimuli, the face-selective posterior superior temporal sulcus (pSTS) in the dorsal visual cortex was significantly correlated with layers in the expression branch of the model, and the anterior inferotemporal cortex (aIT) and anterior fusiform face area (aFFA) in the ventral visual cortex were significantly correlated with layers in the identity branch of the model. Besides, the aFFA and aIT better matched the high layers of the model, while the posterior FFA (pFFA) and occipital facial area (OFA) better matched the middle and early layers of the model, respectively. Overall, our study provides a task-optimization computational model to better understand the neural mechanism underlying face recognition, which suggest that similar to the best-performing model, the human visual system exhibits both dissociated and hierarchical neuroanatomical organization when simultaneously coding facial identity and expression.

Pseudosparse neural coding in the visual system of primates

When measuring sparseness in neural populations as an indicator of efficient coding, an implicit assumption is that each stimulus activates a different random set of neurons. In other words, population responses to different stimuli are, on average, uncorrelated. Here we examine neurophysiological data from four lobes of macaque monkey cortex, including V1, V2, MT, anterior inferotemporal cortex, lateral intraparietal cortex, the frontal eye fields, and perirhinal cortex, to determine how correlated population responses are. We call the mean correlation the pseudosparseness index, because high pseudosparseness can mimic statistical properties of sparseness without being authentically sparse. In every data set we find high levels of pseudosparseness ranging from 0.59–0.98, substantially greater than the value of 0.00 for authentic sparseness. This was true for synthetic and natural stimuli, as well as for single-electrode and multielectrode data. A model indicates that a key variable producing high pseudosparseness is the standard deviation of spontaneous activity across the population. Consistently high values of pseudosparseness in the data demand reconsideration of the sparse coding literature as well as consideration of the degree to which authentic sparseness provides a useful framework for understanding neural coding in the cortex.

Automatic Encoding of a View-Centered Background Image in the Macaque Temporal Lobe.

Perceptual processing along the ventral visual pathway to the hippocampus (HPC) is hypothesized to be substantiated by signal transformation from retinotopic space to relational space, which represents interrelations among constituent visual elements. However, our visual perception necessarily reflects the first person's perspective based on the retinotopic space. To investigate this two-facedness of visual perception, we compared neural activities in the temporal lobe (anterior inferotemporal cortex, perirhinal and parahippocampal cortices, and HPC) between when monkeys gazed on an object and when they fixated on the screen center with an object in their peripheral vision. We found that in addition to the spatially invariant object signal, the temporal lobe areas automatically represent a large-scale background image, which specify the subject's viewing location. These results suggest that a combination of two distinct visual signals on relational space and retinotopic space may provide the first person's perspective serving for perception and presumably subsequent episodic memory.

Forward Processing of Object-Location Association from the Ventral Stream to Medial Temporal Lobe in Nonhuman Primates.

While the hippocampus (HPC) is a prime candidate combining object identity and location due to its strong connections to the ventral and dorsal pathways via surrounding medial temporal lobe (MTL) areas, recent physiological studies have reported spatial information in the ventral pathway and its downstream target in MTL. However, it remains unknown whether the object-location association proceeds along the ventral MTL pathway before HPC. To address this question, we recorded neuronal activity from MTL and area anterior inferotemporal cortex (TE) of two macaques gazing at an object to retain its identity and location in each trial. The results showed significant effects of object-location association at a single-unit level in TE, perirhinal cortex (PRC), and HPC, but not in the parahippocampal cortex. Notably, a clear area difference emerged in the association form: 1) representations of object identity were added to those of subjects' viewing location in TE; 2) PRC signaled both the additive form and the conjunction of the two inputs; and 3) HPC signaled only the conjunction signal. These results suggest that the object and location signals are combined stepwise at TE and PRC each time primates view an object, and PRC may provide HPC with the conjunctional signal, which might be used for encoding episodic memory.

Characteristics of Eye-Position Gain Field Populations Determine Geometry of Visual Space.

We have previously demonstrated differences in eye-position spatial maps for anterior inferotemporal cortex (AIT) in the ventral stream and lateral intraparietal cortex (LIP) in the dorsal stream, based on population decoding of gaze angle modulations of neural visual responses (i.e., eye-position gain fields). Here we explore the basis of such spatial encoding differences through modeling of gain field characteristics. We created a population of model neurons, each having a different eye-position gain field. This population was used to reconstruct eye-position visual space using multidimensional scaling. As gain field shapes have never been well-established experimentally, we examined different functions, including planar, sigmoidal, elliptical, hyperbolic, and mixtures of those functions. All functions successfully recovered positions, indicating weak constraints on allowable gain field shapes. We then used a genetic algorithm to modify the characteristics of model gain field populations until the recovered spatial maps closely matched those derived from monkey neurophysiological data in AIT and LIP. The primary differences found between model AIT and LIP gain fields were that AIT gain fields were more foveally dominated. That is, gain fields in AIT operated on smaller spatial scales and smaller dispersions than in LIP. Thus, we show that the geometry of eye-position visual space depends on the population characteristics of gain fields, and that differences in gain field characteristics for different cortical areas may underlie differences in the representation of space.

A Channel for 3D Environmental Shape in Anterior Inferotemporal Cortex

Inferotemporal cortex (IT) has long been studied as a single pathway dedicated to object vision, but connectivity analysis reveals anatomically distinct channels, through ventral superior temporal sulcus (STSv) and dorsal/ventral inferotemporal gyrus (TEd, TEv). Here, we report a major functional distinction between channels. We studied individual IT neurons in monkeys viewing stereoscopic 3D images projected on a large screen. We used adaptive stimuli to explore neural tuning for 3D abstract shapes ranging in scale and topology from small, closed, bounded objects to large, open, unbounded environments (landscape-like surfaces and cave-like interiors). In STSv, most neurons were more responsive to objects, as expected. In TEd, surprisingly, most neurons were more responsive to 3D environmental shape. Previous studies have localized environmental information to posterior cortical modules. Our results show it is also channeled through anterior IT, where extensive cross-connections between STSv and TEd could integrate object and environmental shape information.

Dimensionality of object representations in monkey inferotemporal cortex.

We have calculated the intrinsic dimensionality of visual object representations in anterior inferotemporal (AIT) cortex, based on responses of a large sample of cells stimulated with photographs of diverse objects. Because dimensionality was dependent on data set size, we determined asymptotic dimensionality as both the number of neurons and number of stimulus image approached infinity. Our final dimensionality estimate was 93 (SD: ± 11), indicating that there is basis set of approximately 100 independent features that characterize the dimensions of neural object space. We believe this is the first estimate of the dimensionality of neural visual representations based on single-cell neurophysiological data. The dimensionality of AIT object representations was much lower than the dimensionality of the stimuli. We suggest that there may be a gradual reduction in the dimensionality of object representations in neural populations going from retina to inferotemporal cortex as receptive fields become increasingly complex.

Distinct neuronal interactions in anterior inferotemporal areas of macaque monkeys during retrieval of object association memory.

In macaque monkeys, the anterior inferotemporal cortex, a region crucial for object memory processing, is composed of two adjacent, hierarchically distinct areas, TE and 36, for which different functional roles and neuronal responses in object memory tasks have been characterized. However, it remains unknown how the neuronal interactions differ between these areas during memory retrieval. Here, we conducted simultaneous recordings from multiple single-units in each of these areas while monkeys performed an object association memory task and examined the inter-area differences in neuronal interactions during the delay period. Although memory neurons showing sustained activity for the presented cue stimulus, cue-holding (CH) neurons, interacted with each other in both areas, only those neurons in area 36 interacted with another type of memory neurons coding for the to-be-recalled paired associate (pair-recall neurons) during memory retrieval. Furthermore, pairs of CH neurons in area TE showed functional coupling in response to each individual object during memory retention, whereas the same class of neuron pairs in area 36 exhibited a comparable strength of coupling in response to both associated objects. These results suggest predominant neuronal interactions in area 36 during the mnemonic processing, which may underlie the pivotal role of this brain area in both storage and retrieval of object association memory.

Recovering stimulus locations using populations of eye-position modulated neurons in dorsal and ventral visual streams of non-human primates

We recorded visual responses while monkeys fixated the same target at different gaze angles, both dorsally (lateral intraparietal cortex, LIP) and ventrally (anterior inferotemporal cortex, AIT). While eye-position modulations occurred in both areas, they were both more frequent and stronger in LIP neurons. We used an intrinsic population decoding technique, multidimensional scaling (MDS), to recover eye positions, equivalent to recovering fixated target locations. We report that eye-position based visual space in LIP was more accurate (i.e., metric). Nevertheless, the AIT spatial representation remained largely topologically correct, perhaps indicative of a categorical spatial representation (i.e., a qualitative description such as “left of” or “above” as opposed to a quantitative, metrically precise description). Additionally, we developed a simple neural model of eye position signals and illustrate that differences in single cell characteristics can influence the ability to recover target position in a population of cells. We demonstrate for the first time that the ventral stream contains sufficient information for constructing an eye-position based spatial representation. Furthermore we demonstrate, in dorsal and ventral streams as well as modeling, that target locations can be extracted directly from eye position signals in cortical visual responses without computing coordinate transforms of visual space.

Shape similarity, better than semantic membership, accounts for the structure of visual object representations in a population of monkey inferotemporal neurons.

The anterior inferotemporal cortex (IT) is the highest stage along the hierarchy of visual areas that, in primates, processes visual objects. Although several lines of evidence suggest that IT primarily represents visual shape information, some recent studies have argued that neuronal ensembles in IT code the semantic membership of visual objects (i.e., represent conceptual classes such as animate and inanimate objects). In this study, we investigated to what extent semantic, rather than purely visual information, is represented in IT by performing a multivariate analysis of IT responses to a set of visual objects. By relying on a variety of machine-learning approaches (including a cutting-edge clustering algorithm that has been recently developed in the domain of statistical physics), we found that, in most instances, IT representation of visual objects is accounted for by their similarity at the level of shape or, more surprisingly, low-level visual properties. Only in a few cases we observed IT representations of semantic classes that were not explainable by the visual similarity of their members. Overall, these findings reassert the primary function of IT as a conveyor of explicit visual shape information, and reveal that low-level visual properties are represented in IT to a greater extent than previously appreciated. In addition, our work demonstrates how combining a variety of state-of-the-art multivariate approaches, and carefully estimating the contribution of shape similarity to the representation of object categories, can substantially advance our understanding of neuronal coding of visual objects in cortex.

In vivo connection imaging revealed distinct feedforward and intrinsic neurons in posterior inferotemporal cortex

We investigated [1] circuits for object recognition in macaque anterior (TE) and posterior inferotemporal cortex (TEO), using a two-step method for in vivo anatomical imaging. In step 1, red fluorescent tracer was injected into TE to reveal and Pre-target patches of feedforward neurons in TEO. In step 2, these were visualized on the cortical surface in vivo, and injected with green fluorescent tracer. Histological processing revealed that patches >500 μm from the injection site in TEO consisted of intermingled green TEO-TE intrinsically projecting neurons and red TEO-to-TE neurons, with only few double-labeled neurons. In contrast, patches near the injection site in TEO contained many double-labeled neurons. Two parallel, spatially intermingled circuits are suggested: (1) TEO neurons having very local intrinsic collaterals and projection to TE (2) TEO neurons projecting more widely in the intrinsic network, but not to TE. These parallel systems might be specialized for, respectively, fast vs. highly processed signals.

Cerebral lateralization of face-sensitive areas in left-handers: Only the FFA does not get it right

Face perception is highly lateralized to the right hemisphere (RH) in humans, as supported originally by observations of face recognition impairment (prosopagnosia) following brain damage. Divided visual field presentations, neuroimaging and event-related potential studies have supported this view. While the latter studies are typically performed in right-handers, the few reported cases of prosopagnosia with unilateral left damage were left-handers, suggesting that handedness may shift or qualify the lateralization of face perception. We tested this hypothesis by recording the whole set of face-sensitive areas in 11 left-handers, using a face-localizer paradigm in functional magnetic resonance imaging (fMRI) (faces, cars, and their phase-scrambled versions). All face-sensitive areas identified (superior temporal sulcus, inferior occipital cortex, anterior infero-temporal cortex, amygdala) were strongly right-lateralized in left-handers, this right lateralization bias being as large as in a population of right-handers (40) tested with the same paradigm (Rossion et al., 2012). The notable exception was the so-called ‘Fusiform face area’ (FFA), an area that was slightly left lateralized in the population of left-handers. Since the left FFA is localized closely to an area sensitive to word form in the human brain (‘Visual Word Form Area’ – VWFA), the enhanced left lateralization of the FFA in left-handers may be due to a decreased competition with the representation of words. The implications for the neural basis of face perception, aetiology of brain lateralization in general, and prosopagnosia are also discussed.

The Extended Functional Neuroanatomy of Emotional Processing Biases for Masked Faces in Major Depressive Disorder

BackgroundMajor depressive disorder (MDD) is associated with a mood-congruent processing bias in the amygdala toward face stimuli portraying sad expressions that is evident even when such stimuli are presented below the level of conscious awareness. The extended functional anatomical network that maintains this response bias has not been established, however.AimsTo identify neural network differences in the hemodynamic response to implicitly presented facial expressions between depressed and healthy control participants.MethodUnmedicated-depressed participants with MDD (n = 22) and healthy controls (HC; n = 25) underwent functional MRI as they viewed face stimuli showing sad, happy or neutral face expressions, presented using a backward masking design. The blood-oxygen-level dependent (BOLD) signal was measured to identify regions where the hemodynamic response to the emotionally valenced stimuli differed between groups.ResultsThe MDD subjects showed greater BOLD responses than the controls to masked-sad versus masked-happy faces in the hippocampus, amygdala and anterior inferotemporal cortex. While viewing both masked-sad and masked-happy faces relative to masked-neutral faces, the depressed subjects showed greater hemodynamic responses than the controls in a network that included the medial and orbital prefrontal cortices and anterior temporal cortex.ConclusionsDepressed and healthy participants showed distinct hemodynamic responses to masked-sad and masked-happy faces in neural circuits known to support the processing of emotionally valenced stimuli and to integrate the sensory and visceromotor aspects of emotional behavior. Altered function within these networks in MDD may establish and maintain illness-associated differences in the salience of sensory/social stimuli, such that attention is biased toward negative and away from positive stimuli.

Defining face perception areas in the human brain: A large-scale factorial fMRI face localizer analysis

A number of human brain areas showing a larger response to faces than to objects from different categories, or to scrambled faces, have been identified in neuroimaging studies. Depending on the statistical criteria used, the set of areas can be overextended or minimized, both at the local (size of areas) and global (number of areas) levels. Here we analyzed a whole-brain factorial functional localizer obtained in a large sample of right-handed participants (40). Faces (F), objects (O; cars) and their phase-scrambled counterparts (SF, SO) were presented in a block design during a one-back task that was well matched for difficulty across conditions. A conjunction contrast at the group level {(F–SF) and (F–O)} identified six clusters: in the pulvinar, inferior occipital gyrus (so-called OFA), middle fusiform gyrus (so-called FFA), posterior superior temporal sulcus, amygdala, and anterior infero-temporal cortex, which were all strongly right lateralized. While the FFA showed the largest difference between faces and cars, it also showed the least face-selective response, responding more to cars than scrambled cars. Moreover, the FFA’s larger response to scrambled faces than scrambled cars suggests that its face-sensitivity is partly due to low-level visual cues. In contrast, the pattern of activation in the OFA points to a higher degree of face-selectivity. A BOLD latency mapping analysis suggests that face-sensitivity emerges first in the right FFA, as compared to all other areas. Individual brain analyses support these observations, but also highlight the large amount of interindividual variability in terms of number, height, extent and localization of the areas responding preferentially to faces in the human ventral occipito-temporal cortex. This observation emphasizes the need to rely on different statistical thresholds across the whole brain and across individuals to define these areas, but also raises some concerns regarding any objective labeling of these areas to make them correspond across individual brains. This large-scale analysis helps understanding the set of face-sensitive areas in the human brain, and encourages in-depth single participant analyses in which the whole set of areas is considered in each individual brain.

Shape effects on reflexive spatial attention are driven by the dorsal stream

In a modified reflexive spatial attention paradigm, when the cue and the target are at the same spatial location, processing of the target is faster when the cue and the target have different shapes compared to same (shape effect). Recent physiological findings suggest distinct population level encoding of shape in ventral versus dorsal cortical visual streams in monkeys. In human observers, we tested whether the effect of shape on reflexive spatial attention could be attributed to ventral and/or dorsal stream encoding of shape. In the modified reflexive spatial attention paradigm, we varied the shapes of the cue and target. Based on data from monkey physiology (Lehky & Sereno, 2007), we selected four pairs of cue and target shapes. In some pairs, cue and target were similarly encoded (similar encoding distance) by a population of cells in the lateral intraparietal cortex, a dorsal stream area, but more dissimilarly encoded (having a greater encoding distance) by a population of cells in the anterior inferotemporal cortex (AIT), a ventral stream area. In other pairs, cue and target were similarly encoded in AIT and had greater dissimilarity in LIP encoding. We found that pairs of cue and target with greater dissimilarity in LIP encoding produced larger and more consistent shape effects up to a cue to target onset asynchrony (CTOA) of 450ms. The shape effects for cue and target pairs with greater dissimilarity in AIT encoding were smaller and inconsistent, suggesting that shape effects in reflexive spatial attention are largely driven by the dorsal stream.

A Head View-Invariant Representation of Gaze Direction in Anterior Superior Temporal Sulcus

SummaryHumans show a remarkable ability to discriminate others' gaze direction, even though a given direction can be conveyed by many physically dissimilar configurations of different eye positions and head views. For example, eye contact can be signaled by a rightward glance in a left-turned head or by direct gaze in a front-facing head. Such acute gaze discrimination implies considerable perceptual invariance. Previous human research found that superior temporal sulcus (STS) responds preferentially to gaze shifts [1], but the underlying representation that supports such general responsiveness remains poorly understood. Using multivariate pattern analysis (MVPA) of human functional magnetic resonance imaging (fMRI) data, we tested whether STS contains a higher-order, head view-invariant code for gaze direction. The results revealed a finely graded gaze direction code in right anterior STS that was invariant to head view and physical image features. Further analyses revealed similar gaze effects in left anterior STS and precuneus. Our results suggest that anterior STS codes the direction of another's attention regardless of how this information is conveyed and demonstrate how high-level face areas carry out fine-grained, perceptually relevant discrimination through invariance to other face features.

Statistics of visual responses in primate inferotemporal cortex to object stimuli

We have characterized selectivity and sparseness in anterior inferotemporal cortex, using a large data set. Responses were collected from 674 monkey inferotemporal cells, each stimulated by 806 object photographs. This 806 × 674 matrix was examined in two ways: columnwise, looking at responses of a single neuron to all images (single-neuron selectivity), and rowwise, looking at the responses of all neurons caused by a single image (population sparseness). Selectivity and sparseness were measured as kurtosis of probability distributions. Population sparseness exceeded single-neuron selectivity, with specific values dependent on the size of the data sample. This difference was principally caused by inclusion, within the population, of neurons with a variety of dynamic ranges (standard deviations of responses over all images). Statistics of large responses were examined by quantifying how quickly the upper tail of the probability distribution decreased (tail heaviness). This analysis demonstrated that population responses had heavier tails than single-neuron responses, consistent with the difference between sparseness and selectivity measurements. Population responses with spontaneous activity subtracted had the heaviest tails, following a power law. The very light tails of single-neuron responses indicate that the critical feature for each neuron is simple enough to have a high probability of occurring within a limited stimulus set. Heavy tails of population responses indicate that there are a large number of different critical features to which different neurons are tuned. These results are inconsistent with some structural models of object recognition that posit that objects are decomposed into a small number of standard features.

On the road to invariant recognition: Explaining tradeoff and morph properties of cells in inferotemporal cortex using multiple-scale task-sensitive attentive learning

Visual object recognition is an essential accomplishment of advanced brains. Object recognition needs to be tolerant, or invariant, with respect to changes in object position, size, and view. In monkeys and humans, a key area for recognition is the anterior inferotemporal cortex (ITa). Recent neurophysiological data show that ITa cells with high object selectivity often have low position tolerance. We propose a neural model whose cells learn to simulate this tradeoff, as well as ITa responses to image morphs, while explaining how invariant recognition properties may arise in stages due to processes across multiple cortical areas. These processes include the cortical magnification factor, multiple receptive field sizes, and top-down attentive matching and learning properties that may be tuned by task requirements to attend to either concrete or abstract visual features with different levels of vigilance. The model predicts that data from the tradeoff and image morph tasks emerge from different levels of vigilance in the animals performing them. This result illustrates how different vigilance requirements of a task may change the course of category learning, notably the critical features that are attended and incorporated into learned category prototypes. The model outlines a path for developing an animal model of how defective vigilance control can lead to symptoms of various mental disorders, such as autism and amnesia.