Monkey Inferotemporal Cortex Research Articles

The role of temporal cortex in the control of attention.

Attention is an indispensable component of active vision. Contrary to the widely accepted notion that temporal cortex processing primarily focusses on passive object recognition, a series of very recent studies emphasize the role of temporal cortex structures, specifically the superior temporal sulcus (STS) and inferotemporal (IT) cortex, in guiding attention and implementing cognitive programs relevant for behavioral tasks. The goal of this theoretical paper is to advance the hypothesis that the temporal cortex attention network (TAN) entails necessary components to actively participate in attentional control in a flexible task-dependent manner. First, we will briefly discuss the general architecture of the temporal cortex with a focus on the STS and IT cortex of monkeys and their modulation with attention. Then we will review evidence from behavioral and neurophysiological studies that support their guidance of attention in the presence of cognitive control signals. Next, we propose a mechanistic framework for executive control of attention in the temporal cortex. Finally, we summarize the role of temporal cortex in implementing cognitive programs and discuss how they contribute to the dynamic nature of visual attention to ensure flexible behavior.

Contribution of receptive field center and surround to repetition suppression in macaque visual area V2

Monkey inferotemporal cortex (ITC) neurons respond with declining strength to repeated presentations of relatively large and complex natural images. This phenomenon – repetition suppression – has often been assumed to arise at the level of ITC because ITC neurons possess the large receptive fields and sophisticated stimulus selectivity necessary for recognizing the image as a repetition. It has been recently discovered, however, that neurons in V2 exhibit repetition suppression under conditions identical to those in studies of ITC. This raises the question: How do V2 neurons, with classical receptive fields encompassing only a small fraction of the image, recognize it as a repetition? One possibility is that they are sensitive to repetition of image content not only in the classical receptive field but also in the receptive field surround. To assess this possibility, we monitored neuronal responses to sequential displays in which we controlled independently the repetition of elements in the classical receptive field and in the surround. Each stimulus consisted of a disk scaled to and centered on the classical receptive field and an adjoining annulus (8° outer diameter). The disks and annuli were taken from different natural scenes. The display on each trial consisted of a prime, a delay, and a probe, each 300 ms in duration. We found that repetition of the central disk was sufficient to produce repetition suppression but that suppression was enhanced by simultaneous repetition of the annulus. Suppression arising from repetition of the annulus occurred in a relatively late phase of the response, in accordance with the idea that it might have been mediated by horizontal connections in V2 and/or feedback from downstream areas. Overall, this work demonstrates that sensitivity to content outside the classical receptive field contributes to the image specificity of repetition suppression observed in V2 neurons to large natural images.

Mechanisms underlying sharpening of visual response dynamics with familiarity.

Experience-dependent modifications of synaptic connections are thought to change patterns of network activities and stimulus tuning with learning. However, only a few studies explored how synaptic plasticity shapes the response dynamics of cortical circuits. Here, we investigated the mechanism underlying sharpening of both stimulus selectivity and response dynamics with familiarity observed in monkey inferotemporal cortex. Broadening the distribution of activities and stronger oscillations in the response dynamics after learning provide evidence for synaptic plasticity in recurrent connections modifying the strength of positive feedback. Its interplay with slow negative feedback via firing rate adaptation is critical in sharpening response dynamics. Analysis of changes in temporal patterns also enables us to disentangle recurrent and feedforward synaptic plasticity and provides a measure for the strengths of recurrent synaptic plasticity. Overall, this work highlights the importance of analyzing changes in dynamics as well as network patterns to further reveal the mechanisms of visual learning.

Unsupervised Temporal Contiguity Experience Does Not Break the Invariance of Orientation Selectivity Across Spatial Frequency.

The images projected onto the retina can vary widely for a single object. Despite these transformations primates can quickly and reliably recognize objects. At the neural level, transformation tolerance in monkey inferotemporal cortex is affected by the temporal contiguity statistics of the visual input. Here we investigated whether temporal contiguity learning also influences the basic feature detectors in lower levels of the visual hierarchy, in particular the independent coding of orientation and spatial frequency (SF) in primary visual cortex. Eight male Long Evans rats were repeatedly exposed to a temporal transition between two gratings that changed in SF and had either the same (control SF) or a different (swap SF) orientation. Electrophysiological evidence showed that the responses of single neurons during this exposure were sensitive to the change in orientation. Nevertheless, the tolerance of orientation selectivity for changes in SF was unaffected by the temporal contiguity manipulation, as observed in 239 single neurons isolated pre-exposure and 234 post-exposure. Temporal contiguity learning did not affect orientation selectivity in V1. The basic filter mechanisms that characterize V1 processing seem unaffected by temporal contiguity manipulations.

Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences

What specific features should visual neurons encode, given the infinity of real-world images and the limited number of neurons available to represent them? We investigated neuronal selectivity in monkey inferotemporal cortex via the vast hypothesis space of a generative deep neural network, avoiding assumptions about features or semantic categories. A genetic algorithm searched this space for stimuli that maximized neuronal firing. This led to the evolution of rich synthetic images of objects with complex combinations of shapes, colors, and textures, sometimes resembling animals or familiar people, other times revealing novel patterns that did not map to any clearsemantic category. These results expand our conception of the dictionary of features encoded in the cortex, and the approach can potentially reveal the internal representations of any system whose input can be captured by a generative model.

Neural Correlate of Visual Familiarity in Macaque Area V2.

Neurons in macaque inferotemporal cortex (ITC) respond less strongly to familiar than to novel images. It is commonly assumed that this effect arises within ITC because its neurons respond selectively to complex images and thus encode in an explicit form information sufficient for identifying a particular image as familiar. However, no prior study has examined whether neurons in low-order visual areas selective for local features also exhibit familiarity suppression. To address this issue, we recorded from neurons in macaque area V2 with semichronic microelectrode arrays while monkeys repeatedly viewed a set of large complex natural images. We report here that V2 neurons exhibit familiarity suppression. The effect develops over several days with a trajectory well fitted by an exponential function with a rate constant of ∼100 exposures. Suppression occurs in V2 at a latency following image onset shorter than its reported latency in ITC.SIGNIFICANCE STATEMENT Familiarity suppression, the tendency for neurons to respond less strongly to familiar than novel images, is well known in monkey inferotemporal cortex. Suppression has been thought to arise in inferotemporal cortex because its neurons respond selectively to large complex images and thus explicitly to encode information sufficient for identifying a particular image as familiar. No previous study has explored the possibility that familiarity suppression occurs even in early-stage visual areas where neurons are selective for simple features in confined receptive fields. We now report that neurons in area V2 exhibit familiarity suppression. This finding challenges our current understanding of information processing in V2 as well as our understanding of the mechanisms that underlie familiarity suppression.

Stimulus Familiarity and Expectation Jointly Modulate Neural Activity in the Visual Ventral Stream.

Prior knowledge about the visual world can change how a visual stimulus is processed. Two forms of prior knowledge are often distinguished: stimulus familiarity (i.e., whether a stimulus has been seen before) and stimulus expectation (i.e., whether a stimulus is expected to occur, based on the context). Neurophysiological studies in monkeys have shown suppression of spiking activity both for expected and for familiar items in object-selective inferotemporal cortex. It is an open question, however, if and how these types of knowledge interact in their modulatory effects on the sensory response. To address this issue and to examine whether previous findings generalize to noninvasively measured neural activity in humans, we separately manipulated stimulus familiarity and expectation while noninvasively recording human brain activity using magnetoencephalography. We observed independent suppression of neural activity by familiarity and expectation, specifically in the lateral occipital complex, the putative human homologue of monkey inferotemporal cortex. Familiarity also led to sharpened response dynamics, which was predominantly observed in early visual cortex. Together, these results show that distinct types of sensory knowledge jointly determine the amount of neural resources dedicated to object processing in the visual ventral stream.

Effect of silhouetting and inversion on view invariance in the monkey inferotemporal cortex.

We effortlessly recognize objects across changes in viewpoint, but we know relatively little about the features that underlie viewpoint invariance in the brain. Here, we set out to characterize how viewpoint invariance in monkey inferior temporal (IT) neurons is influenced by two image manipulations-silhouetting and inversion. Reducing an object into its silhouette removes internal detail, so this would reveal how much viewpoint invariance depends on the external contours. Inverting an object retains but rearranges features, so this would reveal how much viewpoint invariance depends on the arrangement and orientation of features. Our main findings are 1) view invariance is weakened by silhouetting but not by inversion; 2) view invariance was stronger in neurons that generalized across silhouetting and inversion; 3) neuronal responses to natural objects matched early with that of silhouettes and only later to that of inverted objects, indicative of coarse-to-fine processing; and 4) the impact of silhouetting and inversion depended on object structure. Taken together, our results elucidate the underlying features and dynamics of view-invariant object representations in the brain.NEW & NOTEWORTHY We easily recognize objects across changes in viewpoint, but the underlying features are unknown. Here, we show that view invariance in the monkey inferotemporal cortex is driven mainly by external object contours and is not specialized for object orientation. We also find that the responses to natural objects match with that of their silhouettes early in the response, and with inverted versions later in the response-indicative of a coarse-to-fine processing sequence in the brain.

A Balanced Comparison of Object Invariances in Monkey IT Neurons.

Our ability to recognize objects across variations in size, position, or rotation is based on invariant object representations in higher visual cortex. However, we know little about how these invariances are related. Are some invariances harder than others? Do some invariances arise faster than others? These comparisons can be made only upon equating image changes across transformations. Here, we targeted invariant neural representations in the monkey inferotemporal (IT) cortex using object images with balanced changes in size, position, and rotation. Across the recorded population, IT neurons generalized across size and position both stronger and faster than to rotations in the image plane as well as in depth. We obtained a similar ordering of invariances in deep neural networks but not in low-level visual representations. Thus, invariant neural representations dynamically evolve in a temporal order reflective of their underlying computational complexity.

Effective Connectivity Reveals Largely Independent Parallel Networks of Face and Body Patches

The primate brain processes objects in the ventral visual pathway. One object category, faces, is processed in a hierarchical network of interconnected areas along this pathway. It remains unknown whether such an interconnected network is specific for faces or whether there are similar networks for other object classes. For example, the primate inferotemporal cortex also contains a set of body-selective patches, adjacent to the face-selective patches, but it is not known whether these body-selective patches form a similar discretely connected network or whether cross-talk exists between the face- and body-processing systems. To address these questions, we combined fMRI with electrical microstimulation to determine the effective connectivity of fMRI-defined face and body patches. We found that microstimulation of face patches caused increased fMRI activation throughout the face-processing system; microstimulation of the body patches gave similar results restricted to the body-processing system. Critically, our results revealed largely segregated connectivity patterns for the body and face patches. These results suggest that face and body patches form two interconnected hierarchical networks that are largely separated within the monkey inferotemporal cortex. Only a restricted number of voxels were activated by stimulation of both the body and face patches. The latter regions may be important for the integration of face and body information. Our findings are not only essential to advance our understanding of the neural circuits that enable social cognition, but they also provide further insights into the organizing principles of the inferotemporal cortex.

Prediction suppression in monkey inferotemporal cortex depends on the conditional probability between images.

When monkeys view two images in fixed sequence repeatedly over days and weeks, neurons in area TE of the inferotemporal cortex come to exhibit prediction suppression. The trailing image elicits only a weak response when presented following the leading image that preceded it during training. Induction of prediction suppression might depend either on the contiguity of the images, as determined by their co-occurrence and captured in the measure of joint probability P(A,B), or on their contingency, as determined by their correlation and as captured in the measures of conditional probability P(A|B) and P(B|A). To distinguish between these possibilities, we measured prediction suppression after imposing training regimens that held P(A,B) constant but varied P(A|B) and P(B|A). We found that reducing either P(A|B) or P(B|A) during training attenuated prediction suppression as measured during subsequent testing. We conclude that prediction suppression depends on contingency, as embodied in the predictive relations between the images, and not just on contiguity, as embodied in their co-occurrence.

There Is a "U" in Clutter: Evidence for Robust Sparse Codes Underlying Clutter Tolerance in Human Vision.

The neural mechanisms underlying object recognition in cluttered scenes (i.e., containing more than one object) remain poorly understood. Studies have suggested that neural responses to multiple objects correspond to an average of the responses to the constituent objects. Yet, it is unclear how the identities of an unknown number of objects could be disentangled from a confounded average response. Here, we use a popular computational biological vision model to show that averaging-like responses can result from responses of clutter-tolerant neurons to suboptimal stimuli. The model also provides a novel prediction, that human detection ability should show a U-shaped dependency on target-clutter similarity, which is confirmed experimentally, supporting a simple, unifying account of how the brain performs object recognition in clutter.

Dynamics of 3D view invariance in monkey inferotemporal cortex

Rotations in depth are challenging for object vision because features can appear, disappear, be stretched or compressed. Yet we easily recognize objects across views. Are the underlying representations view invariant or dependent? This question has been intensely debated in human vision, but the neuronal representations remain poorly understood. Here, we show that for naturalistic objects, neurons in the monkey inferotemporal (IT) cortex undergo a dynamic transition in time, whereby they are initially sensitive to viewpoint and later encode view-invariant object identity. This transition depended on two aspects of object structure: it was strongest when objects foreshortened strongly across views and were similar to each other. View invariance in IT neurons was present even when objects were reduced to silhouettes, suggesting that it can arise through similarity between external contours of objects across views. Our results elucidate the viewpoint debate by showing that view invariance arises dynamically in IT neurons out of a representation that is initially view dependent.

Coding of relative size in monkey inferotemporal cortex.

We seldom mistake a closer object as being larger, even though its retinal image is bigger. One underlying mechanism could be to calculate the size of the retinal image relative to that of another nearby object. Here we set out to investigate whether single neurons in the monkey inferotemporal cortex (IT) are sensitive to the relative size of parts in a display. Each neuron was tested on shapes containing two parts that could be conjoined or spatially separated. Each shape was presented in four versions created by combining the two parts at each of two possible sizes. In this design, neurons sensitive to the absolute size of parts would show the greatest response modulation when both parts are scaled up, whereas neurons encoding relative size would show similar responses. Our main findings are that 1) IT neurons responded similarly to all four versions of a shape, but tuning tended to be more consistent between versions with proportionately scaled parts; 2) in a subpopulation of cells, we observed interactions that resulted in similar responses to proportionately scaled parts; 3) these interactions developed together with sensitivity to absolute size for objects with conjoined parts but developed slightly later for objects with spatially separate parts. Taken together, our results demonstrate for the first time that there is a subpopulation of neurons in IT that encodes the relative size of parts in a display, forming a potential neural substrate for size constancy.

Aging alters visual processing of objects and shapes in inferotemporal cortex in monkeys.

Visual perception declines with age. Perceptual deficits may originate not only in the optical system serving vision but also in the neural machinery processing visual information. Since homologies between monkey and human vision permit extrapolation from monkeys to humans, data from young, middle aged and old monkeys were analyzed to show age-related changes in the neuronal activity in the inferotemporal cortex, which is critical for object and shape vision. We found an increased neuronal response latency, and a decrease in the stimulus selectivity in the older animals and suggest that these changes may underlie the perceptual uncertainties found frequently in the elderly.

Texture discriminability in monkey inferotemporal cortex predicts human texture perception.

Shape and texture are both important properties of visual objects, but texture is relatively less understood. Here, we characterized neuronal responses to discrete textures in monkey inferotemporal (IT) cortex and asked whether they can explain classic findings in human texture perception. We focused on three classic findings on texture discrimination: 1) it can be easy or hard depending on the constituent elements; 2) it can have asymmetries, and 3) it is reduced for textures with randomly oriented elements. We recorded neuronal activity from monkey inferotemporal (IT) cortex and measured texture perception in humans for a variety of textures. Our main findings are as follows: 1) IT neurons show congruent selectivity for textures across array size; 2) textures that were easy for humans to discriminate also elicited distinct patterns of neuronal activity in monkey IT; 3) texture pairs with asymmetries in humans also exhibited asymmetric variation in firing rate across monkey IT; and 4) neuronal responses to randomly oriented textures were explained by an average of responses to homogeneous textures, which rendered them less discriminable. The reduction in discriminability of monkey IT neurons predicted the reduced discriminability in humans during texture discrimination. Taken together, our results suggest that texture perception in humans is likely based on neuronal representations similar to those in monkey IT.

Image familiarization sharpens response dynamics of neurons in inferotemporal cortex.

Repeated viewing of an image over days and weeks induces a marked reduction in the strength with which neurons in monkey inferotemporal cortex respond to it. The processing advantage that attaches to this reduction is unknown. One possibility is that truncation of the response to a familiar image leaves neurons in a state of readiness to respond to ensuing images and thereby enhances their ability to track rapidly changing displays. We explored this possibility by assessing neuronal responses to familiar and novel images in rapid serial visual displays. Inferotemporal neurons responded more strongly to familiar than to novel images in such displays. The effect was stronger among putative inhibitory neurons than among putative excitatory neurons. A comparable effect occurred at the level of the scalp potential in humans. We conclude that long-term familiarization sharpens the response dynamics of neurons in both monkey and human extrastriate visual cortex.

Dimensionality of object representations in monkey inferotemporal cortex.

We have calculated the intrinsic dimensionality of visual object representations in anterior inferotemporal (AIT) cortex, based on responses of a large sample of cells stimulated with photographs of diverse objects. Because dimensionality was dependent on data set size, we determined asymptotic dimensionality as both the number of neurons and number of stimulus image approached infinity. Our final dimensionality estimate was 93 (SD: ± 11), indicating that there is basis set of approximately 100 independent features that characterize the dimensions of neural object space. We believe this is the first estimate of the dimensionality of neural visual representations based on single-cell neurophysiological data. The dimensionality of AIT object representations was much lower than the dimensionality of the stimuli. We suggest that there may be a gradual reduction in the dimensionality of object representations in neural populations going from retina to inferotemporal cortex as receptive fields become increasingly complex.

Statistical learning of serial visual transitions by neurons in monkey inferotemporal cortex.

If monkeys repeatedly, over the course of weeks, view displays in which two images appear in fixed sequence, then neurons of inferotemporal cortex (ITC) come to exhibit prediction suppression. The response to the trailing image is weaker if it follows the leading image with which it was paired during training than if it follows some other leading image. Prediction suppression is a plausible neural mechanism for statistical learning of visual transitions such as has been demonstrated in behavioral studies of human infants and adults. However, in the human studies, subjects are exposed to continuous sequences in which the same image can be both predicted and predicting and statistical dependency can exist between nonadjacent items. The aim of the present study was to investigate whether prediction suppression in ITC develops under such circumstances. To resolve this issue, we exposed monkeys repeatedly to triplets of images presented in fixed order. The results indicate that prediction suppression can be induced by training not only with pairs of images but also with longer sequences.

Greater sensitivity to nonaccidental than metric changes in the relations between simple shapes in the lateral occipital cortex

Behavioral studies and single cell recordings in monkey inferotemporal cortex have documented greater sensitivity to differences in viewpoint invariant or nonaccidental properties (e.g., straight vs. curved), than metric properties (e.g., degree of curvature) of simple shapes. Are we similarly more sensitive to nonaccidental (NAP) than metric (MP) differences of the relations between objects? We addressed this question with sets of scene triplets that could, from a reference or "Base" scene (e.g., a brick slightly separated from a cylinder), undergo a NAP relational change (e.g., the brick attached to the cylinder) or an MP relational change (e.g., the brick further separated from the cylinder). Critically, both relational variations were matched in physical dissimilarity using pixel energy and the Gabor-jet system, a model of V1 similarity. In an adaptive staircase match-to-sample paradigm, subjects required more than double the presentation durations for detecting differences in MP than NAP relations to achieve equivalent levels of accuracy. In two fMRI experiments, NAP changes consistently produced greater responses in the lateral occipital cortex (LO), but not in earlier retinotopic stages, compared to MP changes, implicating LO as the potential neural locus for where the greater detectability of the differences of NAPs than MPs is made explicit. HMAX, a model of cell tuning in higher-level ventral visual areas, did not consistently reflect the marked NAP advantage witnessed in behavioral performance and in LO responses.