Crossmodal Congruency Task Research Articles

The Crossmodal Congruency Task as a Means to Obtain an Objective Behavioral Measure in the Rubber Hand Illusion Paradigm

The rubber hand illusion (RHI) is a popular experimental paradigm. Participants view touch on an artificial rubber hand while the participants' own hidden hand is touched. If the viewed and felt touches are given at the same time then this is sufficient to induce the compelling experience that the rubber hand is one's own hand. The RHI can be used to investigate exactly how the brain constructs distinct body representations for one's own body. Such representations are crucial for successful interactions with the external world. To obtain a subjective measure of the RHI, researchers typically ask participants to rate statements such as "I felt as if the rubber hand were my hand". Here we demonstrate how the crossmodal congruency task can be used to obtain an objective behavioral measure within this paradigm. The variant of the crossmodal congruency task we employ involves the presentation of tactile targets and visual distractors. Targets and distractors are spatially congruent (i.e. same finger) on some trials and incongruent (i.e. different finger) on others. The difference in performance between incongruent and congruent trials - the crossmodal congruency effect (CCE) - indexes multisensory interactions. Importantly, the CCE is modulated both by viewing a hand as well as the synchrony of viewed and felt touch which are both crucial factors for the RHI. The use of the crossmodal congruency task within the RHI paradigm has several advantages. It is a simple behavioral measure which can be repeated many times and which can be obtained during the illusion while participants view the artificial hand. Furthermore, this measure is not susceptible to observer and experimenter biases. The combination of the RHI paradigm with the crossmodal congruency task allows in particular for the investigation of multisensory processes which are critical for modulations of body representations as in the RHI.

The Crossmodal Congruency Task as a Means to Obtain an Objective Behavioral Measure in the Rubber Hand Illusion Paradigm

The rubber hand illusion (RHI) is a popular experimental paradigm. Participants view touch on an artificial rubber hand while the participants' own hidden hand is touched. If the viewed and felt touches are given at the same time then this is sufficient to induce the compelling experience that the rubber hand is one's own hand. The RHI can be used to investigate exactly how the brain constructs distinct body representations for one's own body. Such representations are crucial for successful interactions with the external world. To obtain a subjective measure of the RHI, researchers typically ask participants to rate statements such as "I felt as if the rubber hand were my hand". Here we demonstrate how the crossmodal congruency task can be used to obtain an objective behavioral measure within this paradigm.The variant of the crossmodal congruency task we employ involves the presentation of tactile targets and visual distractors. Targets and distractors are spatially congruent (i.e. same finger) on some trials and incongruent (i.e. different finger) on others. The difference in performance between incongruent and congruent trials - the crossmodal congruency effect (CCE) - indexes multisensory interactions. Importantly, the CCE is modulated both by viewing a hand as well as the synchrony of viewed and felt touch which are both crucial factors for the RHI.The use of the crossmodal congruency task within the RHI paradigm has several advantages. It is a simple behavioral measure which can be repeated many times and which can be obtained during the illusion while participants view the artificial hand. Furthermore, this measure is not susceptible to observer and experimenter biases. The combination of the RHI paradigm with the crossmodal congruency task allows in particular for the investigation of multisensory processes which are critical for modulations of body representations as in the RHI.

Force feedback facilitates multisensory integration during robotic tool use

The present study investigated the effects of force feedback in relation to tool use on the multisensory integration of visuo-tactile information. Participants learned to control a robotic tool through a surgical robotic interface. Following tool-use training, participants performed a crossmodal congruency task, by responding to tactile vibrations applied to their hands, while ignoring visual distractors superimposed on the robotic tools. In the first experiment it was found that tool-use training with force feedback facilitates multisensory integration of signals from the tool, as reflected in a stronger crossmodal congruency effect with the force feedback training compared to training without force feedback and to no training. The second experiment extends these findings by showing that training with realistic online force feedback resulted in a stronger crossmodal congruency effect compared to training in which force feedback was delayed. The present study highlights the importance of haptic information for multisensory integration and extends findings from classical tool-use studies to the domain of robotic tools. We argue that such crossmodal congruency effects are an objective measure of robotic tool integration and propose some potential applications in surgical robotics, robotic tools, and human-tool interaction.

The effect of limb crossing and limb congruency on multisensory integration in peripersonal space for the upper and lower extremities

The present study investigated how multisensory integration in peripersonal space is modulated by limb posture (i.e. whether the limbs are crossed or uncrossed) and limb congruency (i.e. whether the observed body part matches the actual position of one's limb). This was done separately for the upper limbs (Experiment 1) and the lower limbs (Experiment 2). The crossmodal congruency task was used to measure peripersonal space integration for the hands and the feet. It was found that the peripersonal space representation for the hands but not for the feet is dynamically updated based on both limb posture and limb congruency. Together these findings show how dynamic cues from vision, proprioception, and touch are integrated in peripersonal limb space and highlight fundamental differences in the way in which peripersonal space is represented for the upper and lower extremity.

Partial recovery of visual–spatial remapping of touch after restoring vision in a congenitally blind man

In an initial processing step, sensory events are encoded in modality specific representations in the brain but seem to be automatically remapped into a supra-modal, presumably visual-external frame of reference. To test whether there is a sensitive phase in the first years of life during which visual input is crucial for the acquisition of this remapping process, we tested a single case of a congenitally blind man whose sight was restored after the age of two years. HS performed a tactile temporal order judgment task (TOJ) which required judging the temporal order of two tactile stimuli, one presented to each index finger. In addition, a visual–tactile cross-modal congruency task was run, in which spatially congruent and spatially incongruent visual distractor stimuli were presented together with tactile stimuli. The tactile stimuli had to be localized. Both tasks were performed with an uncrossed and a crossed hand posture. Similar to congenitally blind individuals HS did not show a crossing effect in the tactile TOJ task suggesting an anatomical rather than visual-external coding of touch. In the visual–tactile task, however, external remapping of touch was observed though incomplete compared to sighted controls.These data support the hypothesis of a sensitive phase for the acquisition of an automatic use of visual–spatial representations for coding tactile input. Nonetheless, these representations seem to be acquired to some extent after the end of congenital blindness but seem to be recruited only in the context of visual stimuli and are used with a reduced efficiency.

Multisensory integration across the menstrual cycle

Evidence suggests that spatial processing changes across time in naturally cycling women, which is likely due to neuromodulatory effects of steroid hormones. Yet, it is unknown whether crossmodal spatial processes depend on steroid hormones as well. In the present experiment, the crossmodal congruency task was used to assess visuo-tactile interactions in naturally cycling women, women using hormonal contraceptives and men. Participants adopted either a crossed or uncrossed hands posture. It was tested whether a postural effect of hand crossing on multisensory interactions in the crossmodal congruency task is modulated by women's cycle phase. We found that visuotactile interactions changed according to cycle phase. Naturally cycling women showed a significant difference between the menstrual and the luteal phase for crossed, but not for uncrossed hands postures. The two control groups showed no test sessions effects. Regression analysis revealed a positive relation between estradiol levels and the size of crossmodal congruency effects (CCE), indicating that estradiol seems to have a neuromodulatory effect on posture processing.

Extending the Body to Virtual Tools Using a Robotic Surgical Interface: Evidence from the Crossmodal Congruency Task

The effects of real-world tool use on body or space representations are relatively well established in cognitive neuroscience. Several studies have shown, for example, that active tool use results in a facilitated integration of multisensory information in peripersonal space, i.e. the space directly surrounding the body. However, it remains unknown to what extent similar mechanisms apply to the use of virtual-robotic tools, such as those used in the field of surgical robotics, in which a surgeon may use bimanual haptic interfaces to control a surgery robot at a remote location. This paper presents two experiments in which participants used a haptic handle, originally designed for a commercial surgery robot, to control a virtual tool. The integration of multisensory information related to the virtual-robotic tool was assessed by means of the crossmodal congruency task, in which subjects responded to tactile vibrations applied to their fingers while ignoring visual distractors superimposed on the tip of the virtual-robotic tool. Our results show that active virtual-robotic tool use changes the spatial modulation of the crossmodal congruency effects, comparable to changes in the representation of peripersonal space observed during real-world tool use. Moreover, when the virtual-robotic tools were held in a crossed position, the visual distractors interfered strongly with tactile stimuli that was connected with the hand via the tool, reflecting a remapping of peripersonal space. Such remapping was not only observed when the virtual-robotic tools were actively used (Experiment 1), but also when passively held the tools (Experiment 2). The present study extends earlier findings on the extension of peripersonal space from physical and pointing tools to virtual-robotic tools using techniques from haptics and virtual reality. We discuss our data with respect to learning and human factors in the field of surgical robotics and discuss the use of new technologies in the field of cognitive neuroscience.

I feel who I see: Visual body identity affects visual–tactile integration in peripersonal space

Recent studies have shown the importance of integrating multisensory information in the body representation for constituting self-consciousness. However, one idea that has received only scant attention is that our body representation is also constituted by knowledge of bodily visual characteristics (i.e. 'what I look like'). Here in two experiments we used a full body crossmodal congruency task in which visual distractors were presented on a photograph of the participant, another person, who was either familiar or unfamiliar, or an object. Results revealed that during the 'self-condition' CCEs were enhanced compared to the 'other condition'. The CCE was similar for unfamiliar and familiar others. CCEs for the object condition were significantly smaller. The results show that presentation of an irrelevant image of a body affects multimodal processing and that the effect is enhanced when that image is of the self. The results hold intriguing implications for body representation in social situations.

Early and late activity in somatosensory cortex reflects changes in bodily self-consciousness: An evoked potential study

How can we investigate the brain mechanisms underlying self-consciousness? Recent behavioural studies on multisensory bodily perception have shown that multisensory conflicts can alter bodily self-consciousness such as in the “full body illusion” (FBI) in which changes in self-identification with a virtual body and tactile perception are induced. Here we investigated whether experimental changes in self-identification during the FBI are accompanied by activity changes in somatosensory cortex by recording somatosensory-evoked potentials (SEPs). To modulate self-identification, participants were filmed by a video camera from behind while their backs were stroked, either synchronously (illusion condition) or asynchronously (control condition) with respect to the stroking seen on their virtual body. Tibial nerve SEPs were recorded during the FBI and analysed using evoked potential (EP) mapping. Tactile mislocalisation was measured using the crossmodal congruency task. SEP mapping revealed five sequential periods of brain activation during the FBI, of which two differed between the illusion condition and the control condition. Activation at 30–50ms (corresponding to the P40 component) in primary somatosensory cortex was stronger in the illusion condition. A later activation at ∼110–200ms, likely originating in higher-tier somatosensory regions in parietal cortex, was stronger and lasted longer in the control condition. These data show that changes in bodily self-consciousness modulate activity in primary and higher-tier somatosensory cortex at two distinct processing steps. We argue that early modulations of primary somatosensory cortex may be a consequence of (1) multisensory integration of synchronous vs. asynchronous visuo-tactile stimuli and/or (2) differences in spatial attention (to near or far space) between the conditions. The later activation in higher-tier parietal cortex (and potentially other regions in temporo-parietal and frontal cortex) likely reflects the detection of visuo-tactile conflicts in the asynchronous condition.

Does tool use extend peripersonal space? A review and re-analysis

The fascinating idea that tools become extensions of our body appears in artistic, literary, philosophical, and scientific works alike. In the last 15years, this idea has been reframed into several related hypotheses, one of which states that tool use extends the neural representation of the multisensory space immediately surrounding the hands (variously termed peripersonal space, peri-hand space, peri-cutaneous space, action space, or near space). This and related hypotheses have been tested extensively in the cognitive neurosciences, with evidence from molecular, neurophysiological, neuroimaging, neuropsychological, and behavioural fields. Here, I briefly review the evidence for and against the hypothesis that tool use extends a neural representation of the space surrounding the hand, concentrating on neurophysiological, neuropsychological, and behavioural evidence. I then provide a re-analysis of data from six published and one unpublished experiments using the crossmodal congruency task to test this hypothesis. While the re-analysis broadly confirms the previously reported finding that tool use does not literally extend peripersonal space, the overall effect sizes are small and statistical power is low. I conclude by questioning whether the crossmodal congruency task can indeed be used to test the hypothesis that tool use modifies peripersonal space.

Restricted recovery of external remapping of tactile stimuli after restoring vision in a congenitally blind man

People with surgically removed congenital dense bilateral cataracts offer a natural model of visual deprivation and reafferentation in humans to investigate sensitive periods of multisensory development, for example regarding the recruitment of external or anatomical frames of reference for spatial representation. Here we present a single case (HS; male; 33 years; right-handed), born with congenital dense bilateral cataracts. His lenses were removed at the age of two years, but he received optical aids only at age six. At time of testing, his visual acuity was 30% in the best eye. We performed two tasks, a tactile temporal order judgment task (TOJ) in which two tactile stimuli were presented successively to the index fingers located in the two hemifields, adopting a crossed and uncrossed hand posture. The participant judged as precisely as possible which side was stimulated first. Moreover, we used a crossmodal-congruency task in which a tactile stimulus and an irrelevant visual distracter were presented simultaneously but independently to one of four positions. The participant judged the location (index or thumb) of the tactile stimulus with hands crossed or uncrossed. Speed was emphasized. In contrast to sighted controls, HS did not show a decrement of TOJ performance with hands crossed. Moreover, while the congruency gain was equivalent to sighted controls with uncrossed hands, this effect was significantly reduced with hands crossed. Thus, an external remapping of tactile stimuli still develops after a long phase of visual deprivation. However, remapping seems to be less efficient and to only take place in the context of visual stimuli.

Visuotactile interactions in the congenitally acallosal brain: Evidence for early cerebral plasticity

Studies in patients with an isolated, congenital agenesis of the corpus callosum have documented potentials and limits of brain plasticity. Literature suggests that early reorganization mechanisms can compensate for the absence of the corpus callosum in unisensory tasks that involve interhemispheric transfer. It is unknown, however, how the congenitally acallosal brain processes multisensory information, which presumably requires interhemispheric transfer of modality-specific input. Therefore, we tested five patients with total and one patient with partial agenesis of the corpus callosum in a visuotactile interference task (the “crossmodal congruency task”) with uncrossed and crossed hands and compared their performance to that of 31 healthy controls. We found that congruency effects followed the hands in space not only in healthy, but also in congenitally acallosal individuals. Remarkably, this was also true when patients’ hands crossed the vertical visual meridian and stimuli were presented at the same hand. These results suggest that callosal connectivity is not required for remapping of visuotactile space. We conclude that early brain plasticity allows for compensation of the developmental absence of the corpus callosum in a visuotactile interference task.

Crossmodal congruency measures of lateral distance effects on the rubber hand illusion

Body ownership for an artificial hand and the perceived position of one's own hand can be manipulated in the so-called rubber hand illusion. To induce this illusion, typically an artificial hand is placed next to the participant's body and stroked in synchrony with the real hand, which is hidden from view. Our first aim was to test if the crossmodal congruency task could be used to obtain a measure for the strength of body ownership in the rubber hand illusion. In this speeded location discrimination task participants responded to tactile targets presented to their index or middle finger, while trying to ignore irrelevant visual distracters placed on the artificial hand either on the congruent finger or on the incongruent finger. The difference between performance on congruent and incongruent trials (crossmodal congruency effect, CCE) indicates the amount of multisensory interactions between tactile targets and visual distracters. In order to investigate if changes in body ownership influence the CCE, we manipulated ownership for an artificial hand by synchronous and asynchronous stroking before the crossmodal congruency task (blocked design) in Experiment 1 and during the crossmodal congruency task (interleaved trial-by-trial design) in Experiment 2. Modulations of the CCE by ownership for an artificial hand were apparent in the interleaved trial-by-trial design. These findings suggest that the CCE can be used as an objective measure for body ownership. Secondly, we tested the hypothesis that the lateral spatial distance between the real hand and artificial hand limits the rubber hand illusion. We found no lateral spatial limits for the rubber hand illusion created by synchronous stroking within reaching distances. In conclusion, the sense of ownership seems to be related to modulations of multisensory interactions possibly through peripersonal space mechanisms, and these modulations do not appear to be limited by an increase in distance between artificial hand and real hand.

Interactions of different body parts in peripersonal space: how vision of the foot influences tactile perception at the hand

The body schema, a constantly updated representation of the body and its parts, has been suggested to emerge from body part-specific representations which integrate tactile, visual, and proprioceptive information about the identity and posture of the body. Studies using different approaches have provided evidence for a distinct representation of the visual space approximately 30 cm around the upper body, and predominantly the hands, termed the peripersonal space. In humans, peripersonal space representations have often been investigated with a visual-tactile crossmodal congruency task. We used this task to test if a representation of peripersonal space exists also around the feet, and to explore possible interactions of peripersonal space representations of different body parts. In Experiment 1, tactile stimuli to the hands and feet were judged according to their elevation while visual distractors presented near the same limbs had to be ignored. Crossmodal congruency effects did not differ between the two types of limbs, suggesting a representation of peripersonal space also around the feet. In Experiment 2, tactile stimuli were presented to the hands, and visual distractors were flashed either near the participant's foot, near a fake foot, or in distant space. Crossmodal congruency effects were larger in the real foot condition than in the two other conditions, indicating interactions between the peripersonal space representations of foot and hand. Furthermore, results of all three conditions showed that vision of the stimulated body part, compared to only proprioceptive input about its location, strongly influences crossmodal interactions for tactile perception, affirming the central role of vision in the construction of the body schema.

The selective effect of the image of a hand on visuotactile interactions as assessed by performance on the crossmodal congruency task

Seeing one's own body (either directly or indirectly) can influence visuotactile crossmodal interactions. Previously, it has been shown that even viewing a simple line drawing of a hand can also modulate such crossmodal interactions, as if viewing the picture of a hand somehow primes the representation of one's own hand. However, factors other than the sight of a symbolic picture of a hand may have modulated the crossmodal interactions reported in previous research. In the present study, we examined the crossmodal modulatory effects of viewing five different visual images (photograph of a hand, line drawing of a hand, line drawing of a car, an U-shape, and an ellipse) on tactile performance. Participants made speeded discrimination responses regarding the location of brief vibrotactile targets presented to either the tip or base of their left index finger, while trying to ignore visual distractors presented to either the left or right of central fixation. We compared the visuotactile congruency effects elicited when the five different visual images were presented superimposed over the visual distractors. Participants' tactile discrimination performance was modulated to a significantly greater extent by viewing the photograph of a hand than when viewing the outline drawing of a hand. No such crossmodal congruency effects were reported in any of the other conditions. These results therefore suggest that visuotactile interactions are specifically modulated by the image of the hand rather than just by any simple orientation cues that may be provided by the image of a hand.

Tool use changes multisensory interactions in seconds: evidence from the crossmodal congruency task

Active tool use in human and non-human primates has been claimed to alter the neural representations of multisensory peripersonal space. To date, most studies suggest that a short period of tool use leads to an expansion or elongation of these spatial representations, which lasts several minutes after the last tool use action. However, the possibility that multisensory interactions also change on a much shorter time scale following or preceding individual tool use movements has not yet been investigated. We measured crossmodal (visual-tactile) congruency effects as an index of multisensory integration during two tool use tasks. In the regular tool use task, the participants used one of two tools in a spatiotemporally predictable sequence after every fourth crossmodal congruency trial. In the random tool use task, the required timing and spatial location of the tool use task varied unpredictably. Multisensory integration effects increased as a function of the number of trials since tool use in the regular tool use group, but remained relatively constant in the random tool use group. The spatial distribution of these multisensory effects, however, was unaffected by tool use predictability, with significant spatial interactions found only near the hands and at the tips of the tools. These data suggest that endogenously preparing to use a tool enhances visual-tactile interactions near the tools. Such enhancements are likely due to the increased behavioural relevance of visual stimuli as each tool use action is prepared before execution.

Assessing the influence of schematic drawings of body parts on tactile discrimination performance using the crossmodal congruency task

Seeing one’s own body (either directly or indirectly) can influence visuotactile crossmodal interactions. Recently, it has been shown that even viewing a simple line drawing of a hand can also modulate such crossmodal interactions, as if the picture of the hand somehow corresponds to (or primes) the participants’ own hand. Alternatively, however, it could be argued that the modulatory effects of viewing the picture of a hand on visuotactile interactions might simply be attributed to cognitive processes such as the semantic referral to the relevant body part or to the orientation cues provided by the hand picture instead. In the present study, we evaluated these various different interpretations of the hand picture effect. Participants made speeded discrimination responses to the location of brief vibrotactile targets presented to either the tip or base of their forefinger, while trying to ignore simultaneously-presented visual distractors presented to either side of central fixation. We compared the modulatory effect of the picture of a hand with that seen when the visual distractors were presented next to words describing the tip and base of the forefinger (Experiment 1), or were superimposed over arrows which provided another kind of directional cue (Experiment 2). Tactile discrimination performance was modulated in the hand picture condition, but not in the word or arrow conditions. These results therefore suggest that visuotactile interactions are specifically modulated by the image of the hand rather than by cognitive cues such as simply semantic referral to the relevant body sites and/or any visual orientation cues provided by the picture of a hand.

On the inability to ignore touch when responding to vision in the crossmodal congruency task

We investigated the extent to which people can selectively ignore distracting vibrotactile information when performing a visual task. In Experiment 1, participants made speeded elevation discrimination responses (up vs. down) to a series of visual targets presented from one of two eccentricities on either side of central fixation, while simultaneously trying to ignore task-irrelevant vibrotactile distractors presented independently to the finger (up) vs. thumb (down) of either hand. Participants responded significantly more slowly, and somewhat less accurately, when the elevation of the vibrotactile distractor was incongruent with that of the visual target than when they were presented from the same (i.e., congruent) elevation. This crossmodal congruency effect was significantly larger when the visual and tactile stimuli appeared on the same side of space than when they appeared on different sides, although the relative eccentricity of the two stimuli within the hemifield (i.e., same vs. different) had little effect on performance. In Experiment 2, participants who crossed their hands over the midline showed a very different pattern of crossmodal congruency effects to participants who adopted an uncrossed hands posture. Our results suggest that both the relative external location and the initial hemispheric projection of the target and distractor stimuli contribute jointly to determining the magnitude of the crossmodal congruency effect when participants have to respond to vision and ignore touch.

Investigating the effect of a transparent barrier on the crossmodal congruency effect

A transparent barrier, such as a window, protects us from approaching objects (such as flies), although we can still see the objects coming toward us through the occluder. In the present study, we examined whether this potential dissociation between tactile and visual experience introduced by the presence of a transparent barrier also affects visual-tactile interactions in normal participants, as indexed by performance in the crossmodal congruency task. Participants discriminated the elevation of vibrotactile target stimuli (upper vs lower) presented to the left or right hand while trying to ignore visual distractor lights that could independently be presented from upper or lower locations on either the same or the opposite side. A transparent occluder was placed between the vibrotactile targets and the visual stimuli (the barrier occluded the vibrotactile targets in Experiment 1 and the visual distractors in Experiment 2). Vibrotactile elevation discrimination performance was slower and less accurate when the distractor lights were presented from incongruent locations relative to the target (e.g., lower tactile target with upper distractor light). However, there were no significant differences between performance with and without the transparent occluder present. This pattern of results was replicated in Experiment 3 under conditions where the participants were periodically required to reach around the transparent occluder in order to press buttons placed near to the visual distractors. Taken together, these results support recent neuropsychological evidence [Farne et al. (2003) Int J Psychophysiol 50:51-61] suggesting that visual-tactile interactions in peripersonal space are not necessarily modulated by conscious awareness of the impossibility of our hand being touched by the visual stimuli.

Multisensory contributions to the 3-D representation of visuotactile peripersonal space in humans: evidence from the crossmodal congruency task

In order to determine precisely the location of a tactile stimulus presented to the hand it is necessary to know not only which part of the body has been stimulated, but also where that part of the body lies in space. This involves the multisensory integration of visual, tactile, proprioceptive, and even auditory cues regarding limb position. In recent years, researchers have become increasingly interested in the question of how these various sensory cues are weighted and integrated in order to enable people to localize tactile stimuli, as well as to give rise to the `felt' position of our limbs, and ultimately the multisensory representation of 3-D peripersonal space. We highlight recent research on this topic using the crossmodal congruency task, in which participants make speeded elevation discrimination responses to vibrotactile targets presented to the thumb or index finger, while simultaneously trying to ignore irrelevant visual distractors presented from either the same (i.e., congruent) or a different (i.e., incongruent) elevation. Crossmodal congruency effects (calculated as performance on incongruent − congruent trials) are greatest when visual and vibrotactile stimuli are presented from the same azimuthal location, thus providing an index of common position across different sensory modalities. The crossmodal congruency task has been used to investigate a number of questions related to the representation of space in both normal participants and brain-damaged patients. In this review, we detail the major findings from this research, and highlight areas of convergence with other cognitive neuroscience disciplines.