Visual Guidance Of Action Research Articles

Continuous action with a neurobiologically inspired computational approach reveals the dynamics of selection history.

Everyday perception-action interaction often requires selection of a single goal from multiple possibilities. According to a recent framework of attentional control, object selection is guided not only by the well-established factors of perceptual salience and current goals but also by selection history. Yet, underlying mechanisms linking selection history and visually-guided actions are poorly understood. To examine such interplay and disentangle the impact of target and distractor history on action selection, we employed a priming-of-popout (PoP) paradigm combined with continuous tracking of reaching movements and computational modeling. Participants reached an odd-colored target among homogeneous distractors while we systematically manipulated the sequence of target and distractor colors from one trial to the next. We observed that current reach movements were significantly influenced by the interaction between attraction by the prior target feature and repulsion by the prior distractor feature. With principal component regression, we found that inhibition led by prior distractors influenced reach target selection earlier than facilitation led by the prior target. In parallel, our newly developed computational model validated that current reach target selection can be explained best by the mechanism postulating the preceded impact of previous distractors followed by a previous target. Such converging empirical and computational evidence suggests that the prior selection history triggers a dynamic interplay between target facilitation and distractor inhibition to guide goal-directed action successfully. This, in turn, highlights the necessity of an explicitly integrated approach to determine how visual attentional selection links with adaptive actions in a complex environment.

Melvyn A. Goodale: A visual neuroscientist in action

Mel Goodale has had a multi-faceted career in cognitive neuroscience, principally in the areas of perception, visually-guided action, and visual consciousness. This short article presents a personal reflection on his career from the point of view of a long-time colleague and friend.

Seeing our hand or a tool during visually-guided actions: Different effects on the somatosensory and visual cortices

The processing of proprioceptive information in the context of a conflict between visual and somatosensory feedbacks deteriorates motor performance. Previous studies have shown that seeing one's hand increases the weighting assigned to arm somatosensory inputs. In this light, we hypothesized that the sensory conflict, when tracing the contour of a shape with mirror-reversed vision, will be greater for participants who trace with a stylus seen in their hand (Hand group, n = 17) than for participants who trace with the tip of rod without seen their hand (Tool group, n = 15). Based on this hypothesis, we predicted that the tracing performance with mirror vision will be more deteriorated for the Hand group than for the Tool group, and we predicted a greater gating of somatosensory information for the Hand group to reduce the sensory conflict. The participants of both groups followed the outline of a shape in two visual conditions. Direct vision: the participants saw the hand or portion of a light 40 cm rod directly. Mirror Vision: the hand or the rod was seen through a mirror. We measured tracing performance using a digitizing tablet and the cortical activity with electroencephalography. Behavioral analyses revealed that the tracing performance of both groups was similarly impaired by mirror vision. However, contrasting the spectral content of the cortical oscillatory activity between the Mirror and Direct conditions, we observed that tracing with mirror vision resulted in significantly larger alpha (8–12 Hz) and beta (15–25 Hz) powers in the somatosensory cortex for participants of the Hand group. The somatosensory alpha and beta powers did not significantly differ between Mirror and Direct vision conditions for the Tool group. For both groups, tracing with mirror vision altered the activity of the visual cortex: decreased alpha power for the Hand group, decreased alpha and beta power for the Tool group. Overall, these results suggest that seeing the hand enhanced the sensory conflict when tracing with mirror vision and that the increase of alpha and beta powers in the somatosensory cortex served to reduce the weight assigned to somatosensory information. The increased activity of the visual cortex observed for both groups in the mirror vision condition suggests greater visual processing with increased task difficulty. Finally, the fact that the participants of the Tool group did not show better tracing performance than those of the Hand group suggests that tracing deterioration resulted from a sensorimotor conflict (as opposed to a visuo-proprioceptive conflict).

O hand, where art thou? Mapping hand location across the visual field during common activities

Humans employ visually-guided actions during a myriad of daily activities. These ubiquitous but precise manual actions rely on synergistic work between eye and hand movements. During this close cooperation between hands and eyes, the hands persist in sight in a way which is unevenly distributed across our visual field. One common assertion is that most hand actions occur in the lower visual field (LVF) because the arms are anatomically lower than the head, and objects typically rest on waist-high table surfaces. While experimental work has shown that humans are more efficient at reaching for and grasping targets located below their visual midline (Goodale and Danckert, Exp Brain Res 137:303–308, 2001), there is almost no empirical data detailing where the hands lie in the visual fields during natural hand actions. To build a comprehensive picture of hand location during natural visually guided manual actions, we analyzed data from a large-scale open-access dataset containing 100 h of non-scripted manual object interactions during domestic kitchen tasks filmed from a head-mounted camera. We found a clear vertical visual asymmetry with hands located in the lower visual scene (LVS) in more than 70% of image frames, particularly in ipsilateral space. These findings provide the first direct evidence for the established assumption that hands spend more time in the lower than in the upper visual field (UVF). Further work is required to determine whether this LVF asymmetry differs across the lifespan, in different professions, and in clinical populations.

A neural active inference model of perceptual-motor learning.

The active inference framework (AIF) is a promising new computational framework grounded in contemporary neuroscience that can produce human-like behavior through reward-based learning. In this study, we test the ability for the AIF to capture the role of anticipation in the visual guidance of action in humans through the systematic investigation of a visual-motor task that has been well-explored-that of intercepting a target moving over a ground plane. Previous research demonstrated that humans performing this task resorted to anticipatory changes in speed intended to compensate for semi-predictable changes in target speed later in the approach. To capture this behavior, our proposed "neural" AIF agent uses artificial neural networks to select actions on the basis of a very short term prediction of the information about the task environment that these actions would reveal along with a long-term estimate of the resulting cumulative expected free energy. Systematic variation revealed that anticipatory behavior emerged only when required by limitations on the agent's movement capabilities, and only when the agent was able to estimate accumulated free energy over sufficiently long durations into the future. In addition, we present a novel formulation of the prior mapping function that maps a multi-dimensional world-state to a uni-dimensional distribution of free-energy/reward. Together, these results demonstrate the use of AIF as a plausible model of anticipatory visually guided behavior in humans.

The Influence of Induced Emotions on Distance and Size Perception and on the Grip Scaling During Grasping.

Previous studies have shown that our perception of stimulus properties can be affected by the emotional nature of the stimulus. It is not clear, however, how emotions affect visually-guided actions toward objects. To address this question, we used toy rats, toy squirrels, and wooden blocks to induce negative, positive, and neutral emotions, respectively. Participants were asked to report the perceived distance and the perceived size of a target object resting on top of one of the three emotion-inducing objects; or to grasp the same target object either without visual feedback (open-loop) or with visual feedback (closed-loop) of both the target object and their grasping hand during the execution of grasping. We found that the target object was perceived closer and larger, but was grasped with a smaller grip aperture in the rat condition than in the squirrel and the wooden-block conditions when no visual feedback was available. With visual feedback present, this difference in grip aperture disappeared. These results showed that negative emotion influences both perceived size and grip aperture, but in opposite directions (larger perceived size but smaller grip aperture) and its influence on grip aperture could be corrected by visual feedback, which revealed different effects of emotion to perception and action. Our results have implications on the understanding of the relationship between perception and action in emotional condition, which showed the novel difference from previous theories.

The role of somatosensation in automatic\xa0visuo-motor control: a comparison of congenital and acquired sensory loss

Studies of chronically deafferented participants have illuminated how regaining some motor control after adult-onset loss of proprioceptive and touch input depends heavily on cognitive control. In this study we contrasted the performance of one such man, IW, with KS, a woman born without any somatosensory fibres. We postulated that her life-long absence of proprioception and touch might have allowed her to automate some simple visually-guided actions, something IW appears unable to achieve. We tested these two, and two age-matched control groups, on writing and drawing tasks performed with and without an audio-verbal echoing task that added a cognitive demand. In common with other studies of skilled action, the dual task was shown to affect visuo-motor performance in controls, with less well-controlled drawing and writing, evident as increases in path speed and reduction in curvature and trial duration. We found little evidence that IW was able to automate even the simplest drawing tasks and no evidence for automaticity in his writing. In contrast, KS showed a selective increase in speed of signature writing under the dual-task conditions, suggesting some ability to automate her most familiar writing. We also tested tracing of templates under mirror-reversed conditions, a task that imposes a powerful cognitive planning challenge. Both IW and KS showed evidence of a visuo-motor planning conflict, as did the controls, for shapes with sharp corners. Overall, IW was much faster than his controls to complete tracing shapes, consistent with an absence of visuo-proprioceptive conflict, whereas KS was slower than her controls, especially as the corners became sharper. She dramatically improved after a short period of practice while IW did not. We conclude that KS, who developed from birth without proprioception, may have some visually derived control of movement not under cognitive control, something not seen in IW. This allowed her to automate some writing and drawing actions, but impaired her initial attempts at mirror-tracing. In contrast, IW, who lost somatosensation as an adult, cannot automate these visually guided actions.

The worse eye revisited: Evaluating the impact of asymmetric peripheral vision loss on everyday function

In instances of asymmetric peripheral vision loss (e.g., glaucoma), binocular performance on simple psychophysical tasks (e.g., static threshold perimetry) is well-predicted by the better seeing eye alone. This suggests that peripheral vision is largely ‘better-eye limited’. In the present study, we examine whether this also holds true for real-world tasks, or whether even a degraded fellow eye contributes important information for tasks of daily living. Twelve normally-sighted adults performed an everyday visually-guided action (finding a mobile phone) in a virtual-reality domestic environment, while levels of peripheral vision loss were independently manipulated in each eye (gaze-contingent blur). The results showed that even when vision in the better eye was held constant, participants were significantly slower to locate the target, and made significantly more head- and eye-movements, as peripheral vision loss in the worse eye increased. A purely unilateral peripheral impairment increased response times by up to 25%, although the effect of bilateral vision loss was much greater (>200%). These findings indicate that even a degraded visual field still contributes important information for performing everyday visually-guided actions. This may have clinical implications for how patients with visual field loss are managed or prioritized, and for our understanding of how binocular information in the periphery is integrated.

Soft-assembled human–machine perceptual systems

Cognitive systems are highly adaptable and flexible, such that action and perception capabilities can be achieved with the body in various ways, and incorporate features of the environment and nonbiological tools. Perceptual learning refers to enduring changes to a system’s ability to perceive and respond to environmental stimuli. Here I present an integrative framework for understanding how such capabilities occur in human–machine systems comprising brain–body–tool–environment interactions. Central to this work is the claim that the capacity for high degrees of adaptation, flexibility, and learning are possible because human–machine systems are soft-assembled systems, that is, systems whose material constitution is not rigidly constrained so as to achieve goals via a variety of configurations. I begin by presenting the foundations of the framework on offer: the concepts, methods, and theories of ecological psychology; embodied cognition; dynamical systems theory; and machine intelligence. Next, I apply the framework to the case of visually-guided action. I conclude by explaining how this framework provides the explanatory and investigative tools to understand human–machine perceptual systems as soft-assembled systems that span brains-bodies-tools-environments.

Prehensile kinematics of the marmoset monkey: Implications for the evolution of visually-guided behaviors.

Throughout the primate lineage, there is a wide diversity of prehensile capacity that is thought to stem from individual species foraging patterns. While many studies have explored primates with precise hand grips, such as higher apes, few have considered primates that lack opposition movements. The New World marmoset monkey occupies an intriguing niche, displaying adept control of their hand movements yet their absence of opposable digits results in relatively imprecise grasping actions when compared with those observed in Old World monkeys, apes, and humans. The marmoset monkey offers a unique composition of ancestral primate corticospinal organization combined with skilled hand use to explore the evolution and development of visually-guided actions. In this study, four adult marmosets were trained to perform a series of visually-guided tasks, designed to assess their control over locating and retrieving objects of differing dimensions. Two of these animals received a neonatal lesion of the inferior pulvinar (unilateral), a thalamic nucleus previously demonstrated to be involved in visuomotor development. The kinematics of their reaching and grasping patterns were recorded for offline analysis. Predictive modeling revealed that maximum grip aperture, time to reach peak velocity and hand use were reliable predictors of distinguishing between cohorts. A consistent feature observed across all tasks was that they do not precisely scale their grip according to the dimensions of the target object which may be attributed to their lack of independent digit control. Therefore, the marmoset monkey represents a previously understudied position in the evolution of primate reach and grasp behavior.

The visual guidance of action is not insulated from cognitive interference: A multitasking study on obstacle-avoidance and bisection.

The Perception-Action Model (PAM) considers the visual system to be divided into two streams defined by their specific functions, a ventral stream for vision and a dorsal stream for action. In this study we investigated two behavioral paradigms which according to PAM represent the two contrasting functions of the ventral and dorsal stream, namely bisection and obstacle-avoidance, respectively. It is an assumption of PAM that while ventral stream processing is ultimately linked with processing in other cognitive systems, dorsal stream processing is insulated from cognition. Accordingly it can be expected that a secondary task will interfere with bisection but not with obstacle-avoidance. We tested this prediction using a rapid serial visual presentation task as our secondary task (RSVP). Contrary to expectations we found significant interference for both bisection and obstacle-avoidance. Our findings suggest that dorsal-stream processing is not insulated from cognitive processes.

Congruency between perceptual and conceptual object size modulates visually-guided action

Congruency between perceptual and conceptual object size modulates visually-guided action

Multifractal foundations of visually-guided aiming and adaptation to prismatic perturbation

Visually-guided action of tossing to a target allows examining coordination between mechanical information for maintaining posture while throwing and visual information for aiming. Previous research indicates that relationships between visual and mechanical information persist in tossing behavior long enough for mechanical cues to prompt recall of past visual impressions. Multifractal analysis might model the long-term coordinations among movement components as visual information changes. We asked 32 adult participants (6 female, 25 male, one not conforming to gender binary; aged M=19.77, SD=0.88) to complete an aimed-tossing task in three blocks of ten trials each. Block 1 oriented participants to the task. Participants wore right-shifting goggles in Block 2 and removed them for Block 3. Motion-capture suits collected movement data of the head, hips, and hands. According to regression modeling of tossing performance, multifractality at hand and at hips together supported use of visual information, and adaptation to wearing/removing of goggles depended on multifractality across the hips, head, and hands. Vector-autoregression modeling shows that hip multifractality promoted head multifractality but that hand fluctuations drew on head and hip multifractality. We propose that multifractality could be an information substrate whose spread across the movements systems supports the perceptual coordination for the development of dexterity.

Consciousness and choking in visually-guided actions

Choking under pressure describes the phenomenon of people performing well below their expected standard under circumstances where optimal performance is crucial. One of the prevailing explanations for choking is that pressure increases the conscious attention to the underlying processes of the performer's task execution, thereby disrupting what would normally be a relatively automatic process. However, research on choking has focused mainly on the influence of pressure on motor performance, typically overlooking how it might alter the way that vision is controlled when performing these motor actions. In this article we ask whether the visual component of expert motor-skill execution is susceptible to choking much like the motor component is thought to be. To do so, we draw heavily on empirical findings from studies of sporting expertise, in particular focussing on the role of gaze in three types of visually-guided actions: interceptive actions, aiming tasks, and anticipatory skill. For each of these skills we (i) evaluate the nature of the expert advantage, (ii) discuss the role of consciousness in their control, (iii) examine the potential impact of pressure on task performance, and (iv) consider interventions designed to reduce the likelihood of choking when performing these tasks.

Goal-directed action is automatically biased towards looming motion

It is known that looming motion can capture attention regardless of an observer’s intentions. Real-world behavior, however, frequently involves not just attentional selection, but selection for action. Thus, it is important to understand the impact of looming motion on goal-directed action to gain a broader perspective on how stimulus properties bias human behavior. We presented participants with a visually-guided reaching task in which they pointed to a target letter presented among non-target distractors. On some trials, one of the pre-masks at the location of the upcoming search objects grew rapidly in size, creating the appearance of a “looming” target or distractor. Even though looming motion did not predict the target location, the time required to reach to the target was shorter when the target loomed compared to when a distractor loomed. Furthermore, reach movement trajectories were pulled towards the location of a looming distractor when one was present, a pull that was greater still when the looming motion was on a collision path with the participant. We also contrast reaching data with data from a similarly designed visual search task requiring keypress responses. This comparison underscores the sensitivity of visually-guided reaching data, as some experimental manipulations, such as looming motion path, affected reach trajectories but not keypress measures. Together, the results demonstrate that looming motion biases visually-guided action regardless of an observer’s current behavioral goals, affecting not only the time required to reach to targets but also the path of the observer’s hand movement itself.

Visually-guided interceptive actions performed in virtual environments

Virtual environments are quickly becoming a pervasive part of everyday life, yet little is known about the visuomotor control of actions performed in these conditions. This is particularly the case for interceptive actions, during which performers interact with objects that are seen, but in virtual environments, are not physically present. We hypothesized that the absence of a targeted object would lead to actions being more influenced by illusory context (and hence the result of different visuomotor processing) when compared to actions performed if the object were present. Twelve participants grasped objects of different lengths that were embedded in a control or illusory background (Ponzo illusion; see Ganel, Tanzer, & Goodale, 2008). Crucially, objects were viewed through a half-silvered mirror in two conditions that simulated (i) a natural environment where the objects sat on a surface in front of participants, and (ii) a virtual environment where the objects appeared to be in an identical position, but were actually images produced by the mirror. Optotrak cameras measured the grip aperture of participants. Results showed that, when grasping objects placed against the control background, there was no difference in the scaling of the actions performed in the natural and virtual environments (slope of object size vs. maximum grip aperture; t(11)=-.94, p=.37, two-tailed). In contrast, the illusory background altered the scaling of the grasping actions more in the virtual environment than it did for actions in the natural environment (normalized illusion effect; t(11)=1.92, p=.040, one-tailed). These results suggest that, when compared to natural actions, visually-guided interceptive actions performed in virtual environments are more sensitive to visual illusions. This is consistent with the idea that actions in virtual environments may rely on different (allocentric rather than egocentric) visuomotor processing, and raises doubt about the suitability of virtual environments for testing and training visually-guided actions. Meeting abstract presented at VSS 2014

Eye dominance influences triggering action: The Poffenberger paradigm revisited

Our dominant eye (DE) is the one we unconsciously choose when performing a monocular task. Although it has been recognized for centuries, eye dominance and its behavioral consequences remain poorly understood. Here we used the simple and well-known Poffenberger paradigm (1912) in which participants press a button with the right or left index finger, in reaction to the appearance of a lateralized visual stimulus. By selecting participants according to their DE and handedness, we were able to decipher the impact of eye dominance on visuomotor transformation speed. We show, for the first time, that in right-handers simple reaction times (RT) in response to a lateralized visual target are shorter when it appears in the contralateral visual hemifield with respect to the DE. In left-handers, only those with a right DE exhibit a shorter RT with the left hand and they show no hemifield difference. Additionally, the Poffenberger paradigm has been used to estimate the interhemispheric transfer time (IHTT) in both directions, from the right to the left hemisphere or the reverse, by comparing hand RTs following stimulation of each visual hemifield. The present study demonstrates that this paradigm leads to biased estimations of these directionally considered IHTT and provides an explanation to the often reported IHTT negative values that otherwise appear implausible. These new findings highlight the need to consider eye dominance in studies investigating the neural processes underlying visually-guided actions. More generally, they demonstrate a substantial impact of eye dominance on the neural mechanisms involved in converting visual inputs into motor commands.

Pointing in Visual Periphery: Is DF's Dorsal Stream Intact?

Observations of the visual form agnosic patient DF have been highly influential in establishing the hypothesis that separate processing streams deal with vision for perception (ventral stream) and vision for action (dorsal stream). In this context, DF's preserved ability to perform visually-guided actions has been contrasted with the selective impairment of visuomotor performance in optic ataxia patients suffering from damage to dorsal stream areas. However, the recent finding that DF shows a thinning of the grey matter in the dorsal stream regions of both hemispheres in combination with the observation that her right-handed movements are impaired when they are performed in visual periphery has opened up the possibility that patient DF may potentially also be suffering from optic ataxia. If lesions to the posterior parietal cortex (dorsal stream) are bilateral, pointing and reaching deficits should be observed in both visual hemifields and for both hands when targets are viewed in visual periphery. Here, we tested DF's visuomotor performance when pointing with her left and her right hand toward targets presented in the left and the right visual field at three different visual eccentricities. Our results indicate that DF shows large and consistent impairments in all conditions. These findings imply that DF's dorsal stream atrophies are functionally relevant and hence challenge the idea that patient DF's seemingly normal visuomotor behaviour can be attributed to her intact dorsal stream. Instead, DF seems to be a patient who suffers from combined ventral and dorsal stream damage meaning that a new account is needed to explain why she shows such remarkably normal visuomotor behaviour in a number of tasks and conditions.

“Left Neglected,” but Only in Far Space: Spatial Biases in Healthy Participants Revealed in a Visually Guided Grasping Task

Hemispatial neglect is a common outcome of stroke that is characterized by the inability to orient toward, and attend to stimuli in contralesional space. It is established that hemispatial neglect has a perceptual component, however, the presence and severity of motor impairments is controversial. Establishing the nature of space use and spatial biases during visually guided actions amongst healthy individuals is critical to understanding the presence of visuomotor deficits in patients with neglect. Accordingly, three experiments were conducted to investigate the effect of object spatial location on patterns of grasping. Experiment 1 required right-handed participants to reach and grasp for blocks in order to construct 3D models. The blocks were scattered on a tabletop divided into equal size quadrants: left near, left far, right near, and right far. Identical sets of building blocks were available in each quadrant. Space use was dynamic, with participants initially grasping blocks from right near space and tending to “neglect” left far space until the final stages of the task. Experiment 2 repeated the protocol with left-handed participants. Remarkably, left-handed participants displayed a similar pattern of space use to right-handed participants. In Experiment 3 eye movements were examined to investigate whether “neglect” for grasping in left far reachable space had its origins in attentional biases. It was found that patterns of eye movements mirrored patterns of reach-to-grasp movements. We conclude that there are spatial biases during visually guided grasping, specifically, a tendency to neglect left far reachable space, and that this “neglect” is attentional in origin. The results raise the possibility that visuomotor impairments reported among patients with right hemisphere lesions when working in contralesional space may result in part from this inherent tendency to “neglect” left far space irrespective of the presence of unilateral visuospatial neglect.

Visual control of manual actions: brain mechanisms in typical development and developmental disorders

Some key stages in the development of manual actions have been discussed in this supplement based on the idea of the dorsal cortical stream as the pathway for translating visual information into action control. We argue that visual information, transmitted through specialized visuomotor dorsal-stream modules, is required in the control of manual actions for selecting and attending to the target object of the action, translating visual spatial information into motor programmes and planning a coordinated sequence of actions so as to reach an optimal end-state. In typical development, we illustrate dorsal-stream processing through results on the use of stereoscopic information to guide infants' reaches, and changes in target selection and detailed kinematics of reaches depending on age, object size, and reaching in darkness (when dorsal-stream information rapidly decays). We hypothesize 'dorsal-stream vulnerability' as a widespread feature of neurodevelopmental disorders, such as autism, Williams syndrome, and children born very preterm. Such deficits, identified as abnormal visuomanual actions, are seen in bimanual coordination, visual guidance of action in the 'postbox' task, and failures in motor planning for end-state comfort. We discuss the possible application of these approaches to a wider range of disorders including developmental coordination disorder.