Eye-centered Reference Frame Research Articles

Feature-selective responses in macaque visual cortex follow eye movements during natural vision

In natural vision, primates actively move their eyes several times per second via saccades. It remains unclear whether, during this active looking, visual neurons exhibit classical retinotopic properties, anticipate gaze shifts or mirror the stable quality of perception, especially in complex natural scenes. Here, we let 13 monkeys freely view thousands of natural images across 4.6 million fixations, recorded 883 h of neuronal responses in six areas spanning primary visual to anterior inferior temporal cortex and analyzed spatial, temporal and featural selectivity in these responses. Face neurons tracked their receptive field contents, indicated by category-selective responses. Self-consistency analysis showed that general feature-selective responses also followed eye movements and remained gaze-dependent over seconds of viewing the same image. Computational models of feature-selective responses located retinotopic receptive fields during free viewing. We found limited evidence for feature-selective predictive remapping and no viewing-history integration. Thus, ventral visual neurons represent the world in a predominantly eye-centered reference frame during natural vision.

Psychophysical evidence for the involvement of head/body-centered reference frames in egocentric visuospatial memory: A whole-body roll tilt paradigm.

Accurate memory regarding the location of an object with respect to one's own body, termed egocentric visuospatial memory, is essential for action directed toward the object. Although researchers have suggested that the brain stores information related to egocentric visuospatial memory not only in the eye-centered reference frame but also in the other egocentric (i.e., head- or body-centered or both) reference frames, experimental evidence is scarce. Here, we tested this possibility by exploiting the perceptual distortion of head/body-centered coordinates via whole-body tilt relative to gravity. We hypothesized that if the head/body-centered reference frames are involved in storing the egocentric representation of a target in memory, then reproduction would be affected by this perceptual distortion. In two experiments, we asked participants to reproduce the remembered location of a visual target relative to their head/body. Using intervening whole-body roll rotations, we manipulated the initial (target presentation) and final (reproduction of the remembered location) body orientations in space and evaluated the effect on the reproduced location. Our results showed significant biases of the reproduced target location and perceived head/body longitudinal axis in the direction of the intervening body rotation. Importantly, the amount of error was correlated across participants. These results provide experimental evidence for the neural encoding and storage of information related to egocentric visuospatial memory in the head/body-centered reference frames.

Shifts in the eye-centered frame of reference may underlie saccades, visual perception, and eye-hand coordination.

Conventional, computational theories limit the understanding of how action and perception are controlled. In an alternative scheme, the nervous system controls the values of physical and neurophysiological parameters that predetermine the choice of the spatial frames of reference (FRs) for action and perception. For example, all possible eye positions, Q, can be considered as comprising a spatial FR in which extraocular muscles (EOMs) stabilize gaze directions. The origin or referent point of this FR is a specific, threshold eye position, R, at which EOMs can be quiescent but activated depending on the difference between Q and R. Starting before eye motion, shifts in R cause displacement of the FR and resetting of the stable equilibrium position to which the eyes are forced to move. Rather than corollary discharge, the depiction of visual images integrated across the entire retina in the shifted spatial FR is responsible for remapping visual receptive fields and visual constancy. These suggestions are illustrated in computer models of saccades in the referent control framework in humans and monkeys. The existence of three types of visual RF remapping during saccades is suggested. Properly scaled, shifts in the R underlying a saccade are transmitted to motoneurons of arm muscles to guide reach-to-grasp motion in the same, eye-centered FR. Some predictions of the proposed control scheme have been verified and new tests are suggested. The scheme is applicable to several eye-hand coordination deficits including micrography in Parkinson's disease and explains why vision helps deafferented subjects diminish movement deficits.

Gaze direction influences grasping actions towards unseen, haptically explored, objects

Haptic exploration produces mental object representations that can be memorized for subsequent object-directed behaviour. Storage of haptically-acquired object images (HOIs), engages, besides canonical somatosensory areas, the early visual cortex (EVC). Clear evidence for a causal contribution of EVC to HOI representation is still lacking. The use of visual information by the grasping system undergoes necessarily a frame of reference shift by integrating eye-position. We hypothesize that if the motor system uses HOIs stored in a retinotopic coding in the visual cortex, then its use is likely to depend at least in part on eye position. We measured the kinematics of 4 fingers in the right hand of 15 healthy participants during the task of grasping different unseen objects behind an opaque panel, that had been previously explored haptically. The participants never saw the object and operated exclusively based on haptic information. The position of the object was fixed, in front of the participant, but the subject’s gaze varied from trial to trial between 3 possible positions, towards the unseen object or away from it, on either side. Results showed that the middle and little fingers’ kinematics during reaching for the unseen object changed significantly according to gaze position. In a control experiment we showed that intransitive hand movements were not modulated by gaze direction. Manipulating eye-position produces small but significant configuration errors, (behavioural errors due to shifts in frame of reference) possibly related to an eye-centered frame of reference, despite the absence of visual information, indicating sharing of resources between the haptic and the visual/oculomotor system to delayed haptic grasping.

Prior expectation of objects in space is dependent on the direction of gaze

Many studies of multisensory spatial localization have shown that observers’ responses are well-characterized by Bayesian inference, as localization judgments are influenced not only by the reliability of sensory encoding, but expectations about where things occur in space. Here, we investigate the frame of reference for the prior expectation of objects in space. Using an audiovisual localization task, we systematically manipulate fixation position and evaluate whether this prior is encoded in an eye-centered, head-centered, or hybrid frame of reference. Results show that in a majority of participants, this prior is encoded in an eye-centered frame of reference.

Heading Tuning in Macaque Area V6.

To understand how we successfully navigate our world, it is important to understand which parts of the brain process cues used to perceive our direction of self-motion (i.e., heading). Cortical area V6 has been implicated in heading computations based on human neuroimaging data, but direct measurements of heading selectivity in individual V6 neurons have been lacking. We provide the first demonstration that V6 neurons carry 3D visual heading signals, which are represented in an eye-centered reference frame. In contrast, we found almost no evidence for vestibular heading signals in V6, indicating that V6 is unlikely to contribute to multisensory integration of heading signals, unlike other cortical areas. These findings provide important constraints on the roles of V6 in self-motion perception.

Eye-centered visual receptive fields in the ventral intraparietal area.

The ventral intraparietal area (VIP) processes multisensory visual, vestibular, tactile, and auditory signals in diverse reference frames. We recently reported that visual heading signals in VIP are represented in an approximately eye-centered reference frame when measured using large-field optic flow stimuli. No VIP neuron was found to have head-centered visual heading tuning, and only a small proportion of cells had reference frames that were intermediate between eye- and head-centered. In contrast, previous studies using moving bar stimuli have reported that visual receptive fields (RFs) in VIP are head-centered for a substantial proportion of neurons. To examine whether these differences in previous findings might be due to the neuronal property examined (heading tuning vs. RF measurements) or the type of visual stimulus used (full-field optic flow vs. a single moving bar), we have quantitatively mapped visual RFs of VIP neurons using a large-field, multipatch, random-dot motion stimulus. By varying eye position relative to the head, we tested whether visual RFs in VIP are represented in head- or eye-centered reference frames. We found that the vast majority of VIP neurons have eye-centered RFs with only a single neuron classified as head-centered and a small minority classified as intermediate between eye- and head-centered. Our findings suggest that the spatial reference frames of visual responses in VIP may depend on the visual stimulation conditions used to measure RFs and might also be influenced by how attention is allocated during stimulus presentation.

Bottom-up Retinotopic Organization Supports Top-down Mental Imagery

Finding a path between locations is a routine task in daily life. Mental navigation is often used to plan a route to a destination that is not visible from the current location. We first used functional magnetic resonance imaging (fMRI) and surface-based averaging methods to find high-level brain regions involved in imagined navigation between locations in a building very familiar to each participant. This revealed a mental navigation network that includes the precuneus, retrosplenial cortex (RSC), parahippocampal place area (PPA), occipital place area (OPA), supplementary motor area (SMA), premotor cortex, and areas along the medial and anterior intraparietal sulcus. We then visualized retinotopic maps in the entire cortex using wide-field, natural scene stimuli in a separate set of fMRI experiments. This revealed five distinct visual streams or ‘fingers’ that extend anteriorly into middle temporal, superior parietal, medial parietal, retrosplenial and ventral occipitotemporal cortex. By using spherical morphing to overlap these two data sets, we showed that the mental navigation network primarily occupies areas that also contain retinotopic maps. Specifically, scene-selective regions RSC, PPA and OPA have a common emphasis on the far periphery of the upper visual field. These results suggest that bottom-up retinotopic organization may help to efficiently encode scene and location information in an eye-centered reference frame for top-down, internally generated mental navigation. This study pushes the border of visual cortex further anterior than was initially expected.

Reference Frame of the Ventriloquism Aftereffect

Seeing the image of a newscaster on a television set causes us to think that the sound coming from the loudspeaker is actually coming from the screen. How images capture sounds is mysterious because the brain uses different methods for determining the locations of visual versus auditory stimuli. The retina senses the locations of visual objects with respect to the eyes, whereas differences in sound characteristics across the ears indicate the locations of sound sources referenced to the head. Here, we tested which reference frame (RF) is used when vision recalibrates perceived sound locations. Visually guided biases in sound localization were induced in seven humans and two monkeys who made eye movements to auditory or audiovisual stimuli. On audiovisual (training) trials, the visual component of the targets was displaced laterally by 5-6 degrees. Interleaved auditory-only (probe) trials served to evaluate the effect of experience with mismatched visual stimuli on auditory localization. We found that the displaced visual stimuli induced ventriloquism aftereffect in both humans (approximately 50% of the displacement size) and monkeys (approximately 25%), but only for locations around the trained spatial region, showing that audiovisual recalibration can be spatially specific. We tested the reference frame in which the recalibration occurs. On probe trials, we varied eye position relative to the head to dissociate head- from eye-centered RFs. Results indicate that both humans and monkeys use a mixture of the two RFs, suggesting that the neural mechanisms involved in ventriloquism occur in brain region(s) using a hybrid RF for encoding spatial information.

Parietal Reach Region Encodes Reach Depth Using Retinal Disparity and Vergence Angle Signals

Performing a visually guided reach requires the ability to perceive the egocentric distance of a target in three-dimensional space. Previous studies have shown that the parietal reach region (PRR) encodes the two-dimensional location of frontoparallel targets in an eye-centered reference frame. To investigate how a reach target is represented in three dimensions, we recorded the spiking activity of PRR neurons from two rhesus macaques trained to fixate and perform memory reaches to targets at different depths. Reach and fixation targets were configured to explore whether neural activity directly reflects egocentric distance as the amplitude of the required motor command, which is the absolute depth of the target, or rather the relative depth of the target with reference to fixation depth. We show that planning activity in PRR represents the depth of the reach target as a function of disparity and fixation depth, the spatial parameters important for encoding the depth of a reach goal in an eye centered reference frame. The strength of modulation by disparity is maintained across fixation depth. Fixation depth gain modulates disparity tuning while preserving the location of peak tuning features in PRR neurons. The results show that individual PRR neurons code depth with respect to the fixation point, that is, in eye centered coordinates. However, because the activity is gain modulated by vergence angle, the absolute depth can be decoded from the population activity.

Motor-Related Signals in the Intraparietal Cortex Encode Locations in a Hybrid, rather than Eye-Centered Reference Frame

The reference frame used by intraparietal cortex neurons to encode locations is controversial. Many previous studies have suggested eye-centered coding, whereas we have reported that visual and auditory signals employ a hybrid reference frame (i.e., a combination of head- and eye-centered information) (Mullette-Gillman et al. 2005). One possible explanation for this discrepancy is that sensory-related activity, which we studied previously, is hybrid, whereas motor-related activity might be eye centered. Here, we examined the reference frame of visual and auditory saccade-related activity in the lateral and medial banks of the intraparietal sulcus (areas lateral intraparietal area [LIP] and medial intraparietal area [MIP]) of 2 rhesus monkeys. We recorded from 275 single neurons as monkeys performed visual and auditory saccades from different initial eye positions. We found that both visual and auditory signals reflected a hybrid of head- and eye-centered coordinates during both target and perisaccadic task periods rather than shifting to an eye-centered format as the saccade approached. This account differs from numerous previous recording studies. We suggest that the geometry of the receptive field sampling in prior studies was biased in favor of an eye-centered reference frame. Consequently, the overall hybrid nature of the reference frame was overlooked because the non-eye-centered response patterns were not fully characterized.

Retinotopic remapping through gain-modulated neural fields

Event Abstract Back to Event Retinotopic remapping through gain-modulated neural fields Sebastian Schneegans1* and Gregor Schöner1 1 Ruhr-Universität Bochum, Germany During visual exploration humans continuously make saccades, rapid eye movements that shift the center of gaze to locations of interest, thereby completely altering the retinal image. Two explanations have been proposed to account for the perception of visual stability and for robust spatial memory despite intervening saccades: According to the first one, a representation of the environment in an eye-centered reference frame is remapped whenever a saccade occurs. This hypothesis is supported by findings of neurons in macaque parietal and frontal cortex whose receptive fields appear to shift transiently in accordance with the metrics of an impending saccade, starting before the actual eye movement is executed [1]. The second hypothesis states that a more gaze-invariant representation (head- or body-centered) is constructed from the eye-centered input, facilitating trans-saccadic integration and memory. This reference frame transformation is hypothesized to be based on so-called gain-modulated neurons found in posterior parietal cortex, which have visual receptive fields in eye-centered coordinates, but whose overall response strength is modulated by current eye position [2]. We show how these two hypotheses can be combined in a single mechanism. As a framework we use dynamic neural fields, which provide a biologically plausible architecture to model neural activation patterns at population level [3]. The key element of the mechanism is a high-dimensional transformation field, which receives eye position and retinotopic visual input and projects to a head-centered map. The units of this field show the same overall response pattern as the gain-modulated neurons, and the architecture is analogous to previous implementations of the second hypothesis [4]. Unlike previous approaches, we aim to capture the time course of neural activation patterns under changing inputs, taking into account feedforward and feedback connections between the different maps. This connectivity produces distributed representations of perceptual and memory items, stabilized by mutual excitation between the transformation field and the head-centered map. Crucially, a retinotopic remapping as claimed by the first hypothesis emerges directly from this architecture: Whenever the eye position changes, the combined inputs to the transformation field lead to a shift of activity with respect to the eye-centered reference frame. To make this remapping predictive, the representation of eye position itself is updated by a corollary discharge signal (which has been shown to be a prerequisite for the remapping [5]) using a second, analogous mechanism.

Dorsal Premotor Neurons Encode the Relative Position of the Hand, Eye, and Goal during Reach Planning

When reaching to grasp an object, we often move our arm and orient our gaze together. How are these movements coordinated? To investigate this question, we studied neuronal activity in the dorsal premotor area (PMd) and the medial intraparietal area (area MIP) of two monkeys while systematically varying the starting position of the hand and eye during reaching. PMd neurons encoded the relative position of the target, hand, and eye. MIP neurons encoded target location with respect to the eye only. These results indicate that whereas MIP encodes target locations in an eye-centered reference frame, PMd uses a relative position code that specifies the differences in locations between all three variables. Such a relative position code may play an important role in coordinating hand and eye movements by computing their relative position.

Different memory types for generating saccades at different stages of learning

It has been proposed that positional memories encoded in different types of reference frame are used for the reaching hand movement in different stages of learning. However, the types of reference frame employed for generating behavior at each stage of learning remain unclear, particularly for saccades. To examine the types of reference frame for target positions, we analyzed the saccade of monkeys performing an oculomotor task. The task required the animal to make a learning-based saccade to one of the eight landmark positions specified by the color of a fixation point. According to the color of the fixation point, the target landmark position was fixed in all experimental blocks (FIX trial) or altered to other landmark positions block by block (ALTER trial). Although the monkeys learned the target landmark position in both the FIX and the ALTER trials, once the landmarks became invisible, the success rate remained high only in the FIX trials. These results suggest that the target position was learned on the basis of the landmark positions in the early stage of learning. However, the memory of the target position in space was formed after sufficient training. When the fixation point was shifted horizontally by 5° and the landmarks were invisible, the saccades in the ALTER trials were made to the normal target landmark position whereas those in the FIX trials were made to the point approximately 5° shifted horizontally from the normal target landmark position. These results suggest that the target position in space was initially represented in the head-centered or world-centered reference frame and then in the eye-centered reference frame. Analysis of saccade end-points indicated that a kinematically similar saccade was generated for each FIX trial. These results showed that memories encoded by different reference frames were formed to generate a saccade while the saccade toward the same target was repeatedly executed.

The posterior parietal cortex: Sensorimotor interface for the planning and online control of visually guided movements

We present a view of the posterior parietal cortex (PPC) as a sensorimotor interface for visually guided movements. Special attention is given to the role of the PPC in arm movement planning, where representations of target position and current hand position in an eye-centered frame of reference appear to be mapped directly to a representation of motor error in a hand-centered frame of reference. This mapping is direct in the sense that it does not require target position to be transformed into intermediate reference frames in order to derive a motor error signal in hand-centered coordinates. Despite being direct, this transformation appears to manifest in the PPC as a gradual change in the functional properties of cells along the ventro–dorsal axis of the superior parietal lobule (SPL), i.e. from deep in the sulcus to the cortical surface. Possible roles for the PPC in context dependent coordinate transformations, formation of intrinsic movement representations, and in online control of visually guided arm movements are also discussed. Overall these studies point to the emerging view that, for arm movements, the PPC plays a role not only in the inverse transformations required to convert sensory information into motor commands but also in ‘forward’ transformations as well, i.e. in integrating sensory input with previous and ongoing motor commands to maintain a continuous estimate of arm state that can be used to update present and future movement plans. Critically, this state estimate appears to be encoded in an eye-centered frame of reference.

Frame-Up. Focus on “Eye-Centered, Head-Centered, and Complex Coding of Visual and Auditory Targets in the Intraparietal Sulcus”

As neuroscientists, the world would be simpler if the frame of reference for spatial information in the brain reflected either the sensory apparatus from which the signals were derived or the motor apparatus toward which the signals were aimed. In this simple world, the neural correlate of a visual stimulus would reflect where on the retina the stimulus had appeared, while the neural correlate of an auditory stimulus would reflect where the sound source was relative to the ears. An area involved in coding arm movements would reflect the goal location relative to current arm position. Unfortunately, the world is not so simple. In this issue of the Journal of Neurophysiology (p. 2331–2352), Mullette-Gillman and colleagues (2005), recording from neurons on both banks of the intraparietal sulcus (IPS), show that individual neurons represent the spatial locations of visual and auditory stimuli similarly and that many neurons use idiosyncratic frames of reference that are neither head-centered nor eye-centered. Similar findings were just reported by Schlack and colleagues (2005) in the IPS fundus. These findings are part of a shift toward a new view of frames of reference in the brain.

Eye-Centered, Head-Centered, and Complex Coding of Visual and Auditory Targets in the Intraparietal Sulcus

The integration of visual and auditory events is thought to require a joint representation of visual and auditory space in a common reference frame. We investigated the coding of visual and auditory space in the lateral and medial intraparietal areas (LIP, MIP) as a candidate for such a representation. We recorded the activity of 275 neurons in LIP and MIP of two monkeys while they performed saccades to a row of visual and auditory targets from three different eye positions. We found 45% of these neurons to be modulated by the locations of visual targets, 19% by auditory targets, and 9% by both visual and auditory targets. The reference frame for both visual and auditory receptive fields ranged along a continuum between eye- and head-centered reference frames with approximately 10% of auditory and 33% of visual neurons having receptive fields that were more consistent with an eye- than a head-centered frame of reference and 23 and 18% having receptive fields that were more consistent with a head- than an eye-centered frame of reference, leaving a large fraction of both visual and auditory response patterns inconsistent with both head- and eye-centered reference frames. The results were similar to the reference frame we have previously found for auditory stimuli in the inferior colliculus and core auditory cortex. The correspondence between the visual and auditory receptive fields of individual neurons was weak. Nevertheless, the visual and auditory responses were sufficiently well correlated that a simple one-layer network constructed to calculate target location from the activity of the neurons in our sample performed successfully for auditory targets even though the weights were fit based only on the visual responses. We interpret these results as suggesting that although the representations of space in areas LIP and MIP are not easily described within the conventional conceptual framework of reference frames, they nevertheless process visual and auditory spatial information in a similar fashion.

Multiple frames of reference for pointing to a remembered target

Pointing with an unseen hand to a visual target that disappears prior to movement requires maintaining a memory representation about the target location. The target location can be transformed either into a hand-centered frame of reference during target presentation and remembered under that form, or remembered in terms of retinal and extra-retinal cues and transformed into a body-centered frame of reference before movement initiation. The main goal of the present study was to investigate whether the target is stored in memory in an eye-centered frame, a hand-centered frame or in both frames of reference concomitantly. The task was to locate, memorize, and point to a target in a dark environment. Hand movement was not visible. During the recall delay, participants were asked to move their hand or their eyes in order to disrupt the memory representation of the target. Movement of the eyes during the recall delay was expected to disrupt an eye-centered memory representation whereas movement of the hand was expected to disrupt a hand-centered memory representation by increasing movement variability to the target. Variability of movement amplitude and direction was examined. Results showed that participants were more variable on the directional component of the movement when required to move their hand during recall delay. On the contrary, moving the eyes caused an increase in variability only in the amplitude component of the pointing movement. Taken together, these results suggest that the direction of the movement is coded and remembered in a frame of reference linked to the arm, whereas the amplitude of the movement is remembered in an eye-centered frame of reference.

Eye Position Affects Activity in Primary Auditory Cortex of Primates

Background: Neurons in primary auditory cortex are known to be sensitive to the locations of sounds in space, but the reference frame for this spatial sensitivity has not been investigated. Conventional wisdom holds that the auditory and visual pathways employ different reference frames, with the auditory pathway using a head-centered reference frame and the visual pathway using an eye-centered reference frame. Reconciling these discrepant reference frames is therefore a critical component of multisensory integration.Results: We tested the reference frame of neurons in the auditory cortex of primates trained to fixate visual stimuli at different orbital positions. We found that eye position altered the activity of about one third of the neurons in this region (35 of 113, or 31%). Eye position affected not only the responses to sounds (26 of 113, or 23%), but also the spontaneous activity (14 of 113, or 12%). Such effects were also evident when monkeys moved their eyes freely in the dark. Eye position and sound location interacted to produce a representation for auditory space that was neither head- nor eye-centered in reference frame.Conclusions: Taken together with emerging results in both visual and other auditory areas, these findings suggest that neurons whose responses reflect complex interactions between stimulus position and eye position set the stage for the eventual convergence of auditory and visual information.

Systematic errors of planar arm movements provide evidence for space categorization effects and interaction of multiple frames of reference.

Healthy humans performed arm movements in a horizontal plane, from an initial position toward remembered targets, while the movement and the targets were projected on a vertical computer monitor. We analyzed the mean error of movement endpoints and we observed two distinct systematic error patterns. The first pattern resulted in the clustering of movement endpoints toward the diagonals of the four quadrants of an imaginary circular area encompassing all target locations (oblique effect). The second pattern resulted in a tendency of movement endpoints to be closer to the body or equivalently lower than the actual target positions on the computer monitor (y-effect). Both these patterns of systematic error increased in magnitude when a time delay was imposed between target presentation and initiation of movement. In addition, the presence of a stable visual cue in the vicinity of some targets imposed a novel pattern of systematic errors, including minimal errors near the cue and a tendency for other movement endpoints within the cue quadrant to err away from the cue location. A pattern of systematic errors similar to the oblique effect has already been reported in the literature and is attributed to the subject's conceptual categorization of space. Given the properties of the errors in the present work, we discuss the possibility that such conceptual effects could be reflected in a broad variety of visuomotor tasks. Our results also provide insight into the problem of reference frames used in the execution of these aiming movements. Thus, the oblique effect could reflect a hand-centered reference frame while the y-effect could reflect a body or eye-centered reference frame. The presence of the stable visual cue may impose an additional cue-centered (allocentric) reference frame.