Intersensory Integration Research Articles

Movement-related tactile gating in blindness

When we perform an action, self-elicited movement induces suppression of somatosensory information to the cortex, requiring a correct motor-sensory and inter-sensory (i.e. cutaneous senses, kinesthesia, and proprioception) integration processes to be successful. However, recent works show that blindness might impact some of these elements. The current study investigates the effect of movement on tactile perception and the role of vision in this process. We measured the velocity discrimination threshold in 18 sighted and 18 blind individuals by having them perceive a sequence of two movements and discriminate the faster one in passive and active touch conditions. Participants’ Just Noticeable Difference (JND) was measured to quantify their precision. Results showed a generally worse performance during the active touch condition compared to the passive. In particular, this difference was significant in the blind group, regardless of the blindness duration, but not in the sighted one. These findings suggest that the absence of visual calibration impacts motor-sensory and inter-sensory integration required during movement, diminishing the reliability of tactile signals in blind individuals. Our work spotlights the need for intervention in this population and should be considered in the sensory substitution/reinforcement device design.

Proprioceptive-Visual Integration and Embodied Cognition: A Developmental Perspective.

The present study addressed whether a spatial matching task relying on inter-sensory judgments might be influenced by the transition between egocentric and decentered mode of thinking. Thirty-six children aged 5, 7, or 9 years performed a paramedian correspondence task with conditions requiring inter-sensory integration (visual-proprioceptive) and a problem-solving task. The visual information was given through a mirror that perturbed the sensory judgments in the paramedian correspondence task. The results did not corroborate the hypotheses; nevertheless, the increasing importance of proprioception in the perceptual judgments for older children suggests that although younger children may be body centered (egocentric mode of thinking), older children seemed more able to use body to mediate their perceptual judgments.

Again, knowledge of common source fails to promote visual-haptic integration.

The present study tested whether knowledge of a common source of conflicting visual-haptic stimulation promotes intersensory integration. 40 undergraduates manually felt the size of a square while viewing it through a lens that minified its visual size by half. Participants, however, could experience the haptic and the visual stimulation as emanating from either a common source or different sources. Their subsequent matches of the perceived size were biased by the felt size of the square, irrespective of whether they experienced the intersensory stimulation as coming from one or two sources. These results strengthen previous findings suggesting that the integration of sensory discordant information depends on bottom-up rather than on top-down processes.

A “Unity Assumption” Does Not Promote Intersensory Integration

An account of intersensory integration is premised on knowing that different sensory inputs arise from the same object. Could, however, the combination of the inputs be impaired although the "unity assumption" holds? Forty observers viewed a square through a minifying (50%) lens while they simultaneously touched the square. Half could see and half could not see their haptic explorations of the square. Both groups, however, had reason to believe that they were touching and viewing the same square. Subsequent matches of the inspected square were mutually biased by touch and vision when the exploratory movements were visible. However, the matches were biased in the direction of the square's haptic size when observers could not see their exploratory movements. This impaired integration without the visible haptic explorations suggests that the unity assumption alone is not enough to promote intersensory integration.

Bimodal Stimuli Do Not Always Facilitate Infants' Rule Learning

Seven-month-old infants possess the ability to extract and generalize abstract rules from unimodal sequences of auditory syllables such as ABA and ABB whereas five-month-old infants are facilitated in their learning of these rules by bimodal sequences. In the current study, we investigated whether bimodal stimuli could also help infants acquire what we consider a more difficult abstract rule: AAB. Nineteen 8- to 9-month-old infants were habituated to the AAB rule with simultaneous visual shapes and speech sounds. They were later tested on their ability to discriminate this sequence from ABA and ABB with their looking times measured after each stimulus presentation at test periods. There was no significant difference in looking times amongst the different sequences, suggesting infants were not able to formulate an AAB rule different from those for ABA and ABB. Older infants' inability to utilize extra sensory information to acquire an abstract rule in which younger ones were ready to learn from unimodal stimulus (speech) was unexpected. It is unclear what leads to the failure of rule learning from bimodal stimuli, but possible explanations including perceptual narrowing in the inter-sensory integration and the difference in sensory representation of abstract rules with different difficulties are discussed.

Conflict with vision diminishes proprioceptive adaptation to muscle vibration

Muscle vibration excites muscle spindles and creates illusory movement of a body part in a blindfolded individual. It is followed by an aftereffect, an illusion of return movement when vibration stops. The aftereffect reflects adaptation in the proprioceptive system. This adaptation is susceptible to attentional manipulations (Seizova-Cajic and Azzi in Exp Brain Res 203(1):213-219, 2010), but it is not known whether it is open to cross-modal influences unaided by those manipulations. We attempted to answer this question by allowing vision of the vibrated, stationary arm. We asked our participants (n=20) to retain focus on the feeling of movement. They reported any illusory movement during 60-s biceps vibration (at 90Hz), as well as following its offset, when vision of the arm was removed. During vibration, the proprioceptive movement illusion persisted, although the stationary arm was visible, but its duration and strength were much reduced in comparison with the no-vision condition. The movement aftereffect, experienced in total darkness following vibration offset, was also substantially weaker. The results show that proprioceptive adaptation is strongly modulated by vision. We propose that two processes contribute: perceptual (cross-modal binding with conflicting vision reduces the proprioceptive movement signal) and attentional (view of a stationary arm distracts from the proprioceptive movement signal). Our finding that during vibration, participants felt movement in the arm they could see, which was stationary, shows that cross-modal binding partially failed. This happened because the two percepts were too discrepant. However, only one-the visual-appeared real, and we argue that such an outcome is consistent with general principles of intersensory integration.

Visual cues as a surrogate for tactile feedback during robotic‐assisted laparoscopic prostatectomy: posterolateral margin rates in 1340 consecutive patients

To analyse consecutive cases of robotic-assisted laparoscopic prostatectomy (RALP), present the incidence of nerve-sparing-related positive surgical margins (SM+), include visual cues that might assist in smoothly changing to the robotic platform, and discuss the scientific rationale for 'intersensory integration' which might explain the 'reverse Braille' phenomenon, i.e. the ability to feel when vision is greatly enhanced, as the lack of tactile feedback during RALP is often cited as a disadvantage of robotic surgery, interfering with a surgeon's ability to make intraoperative oncological decisions. Data from 1340 consecutive patients undergoing RALP from one institution were analysed and trends for positive posterolateral SM+ (PLSM+) were correlated with oncological variables before and after RALP. A sample of patient slides were reviewed by a extramural pathologist. Multivariate regression modelling was used to compare the projected rates of PLSM+ vs the actual rate, given the effect of a conscious effort to use visual cues. Finally, video recordings of the procedure were systematically reviewed and correlated with anatomical and histopathological images in an integrated session involving the surgeon and the pathology team. The incidence of PLSM+ was 2.1%, which gradually declined to 1.0% in the last 100 patients. The reduction in PLSM+ occurred despite an increased rate of high-risk tumours operated on during this period. Forecasting analysis showed that the actual PLSM+ rate declined by half in the most recent 1000 patients, due to an integrated effort involving the use of visual cues during surgery. The following visual cues were considered important; appreciation of periprostatic (lateral prostatic) fascial compartments; colour and texture of the tissue; periprostatic veins as a landmark for athermal dissection; signs of inflammation; and a freely separating bloodless plane showing loose shiny areolar tissue. Adapting to the robotic platform is easy and there is no compromise of the oncological safety of this procedure. Experienced surgeons can use visual cues to assist during nerve-sparing RALP and achieve low PLSM+ rates.

Causal inference in audiovisual speech: Comment on “Crossmodal influences on visual perception” by L. Shams

Shams and Kim (S&K) [1], argue that visual perception can be affected by sound and touch, and that such alterations can occur at early stages of processing. This view is by now well-documented, but here I would like to draw some parallels to another domain – the processing of audiovisual speech – that allows one to make interesting generalizations to the auditory domain. It is well-known that the identification of an auditory utterance can be strongly biased by concurrent lipread information and that the lipread information may even change “sensory-specific” auditory cortex [2,3]. Less well-known is that perception of auditory speech is – like visual perception – also highly adaptive even in the mature brain. Recent research has demonstrated that adult listeners readily use lipread information to adjust the phonetic boundary between two speech categories [4]: For example, an auditory ambiguous sound halfway between /b/ and /d/ is perceived as /b/ if the sound was on a previous occasion combined with lipread /b/, but as /d/ if the same ambiguous sound was on the previous occasion combined with lipread /d/. Lipread information thus can serve as a ‘teacher’ for auditory perception. Audiovisual speech also provides a unique test case to examine the problem of causal inference without the need to change potentially important characteristic of the stimulus. As S&K rightly point out, multisensory integration only makes sense if the sensory signals are caused by the same object. This is a non-trivial inference problem for the brain as it typically does not have a clue about the causal structure of an event. S&K adopt a Bayesian approach to examine causal inference and interactions in the audiovisual spatial and numerical domain. We used another procedure to address the same question for speech, namely by creating Sine Wave Speech’ (SWS) [5]. In SWS the spectral richness of speech is reduced to a few sinusoids, and naive listeners typically perceive the sounds as non-speech whistles. Only when informed that the sounds are actually derived from an utterance, listeners perceive the sound as speech and then cannot switch back to non-speech mode again. The interesting part of this is that only listeners in speech mode are affected by lipread information, – both immediate identification and adaptation aftereffects – while listeners in non-speech mode do not show these intersensory integration effects. This finding demonstrates the causal inference problem in the clearest way: Lipread information affects sound identification only if the two information sources are assigned to the same event: namely an articulatory gesture.

The decline of cross-species intersensory perception in human infants: Underlying mechanisms and its developmental persistence

The current study investigated the mechanisms underlying the developmental decline in cross-species intersensory matching first reported by Lewkowicz and Ghazanfar [Lewkowicz, D.J., & Ghazanfar, A.A., (2006). The decline of cross-species intersensory perception in human infants. Proc. Natl. Acad. Sci. U. S. A. 103(17), 6771–6774] and whether the decline persists into later development. Experiment 1 investigated whether infants can match monkey vocalizations to asynchronously presented faces and found that neither 4–6 nor 8–10 month-old infants did. Experiment 1 also assessed whether a visual processing deficit may account for the developmental decline in cross-species matching and indicated that it does not because both age groups discriminated silent monkey calls. Experiment 2 investigated whether an auditory processing deficit may account for the decline and indicated that it does not because 8–10 month-old infants discriminated the acoustic versions of the calls. Finally, Experiment 3 asked whether the developmental decline persists into later development by testing cross-species intersensory matching in 12- and 18-month-old infants and showed that it does because neither age group made intersensory matches. Together, these results bolster prior evidence of a decline in cross-species intersensory integration in early human development and shed new light on the mechanisms underlying it.

Neural basis of aging: the penetration of cognition into action control.

Although functional imaging studies have frequently examined age-related changes in neural recruitment during cognitive tasks, much less is known about such changes during motor performance. In the present study, we used functional magnetic resonance imaging to investigate age-related changes in cyclical hand and/or foot movements across different degrees of complexity. Right-handed volunteers (11 young, 10 old) were scanned while performing isolated flexion-extension movements of the right wrist and foot as well as their coordination, according to the "easy" isodirectional and "difficult" nonisodirectional mode. Findings revealed activation of a typical motor network in both age groups, but several additional brain areas were involved in the elderly. Regardless of the performed motor task, the elderly exhibited additional activation in areas involved in sensory processing and integration, such as contralateral anterior insula, frontal operculum, superior temporal gyrus, supramarginal gyrus, secondary somatosensory area, and ipsilateral precuneus. Age-related activation differences during coordination of both segments were additionally observed in areas reflecting increased cognitive monitoring of motor performance, such as the pre-supplementary motor area, pre-dorsal premotor area, rostral cingulate, and prefrontal cortex. In the most complex coordination task, the elderly exhibited additional activation in anterior rostral cingulate and dorsolateral prefrontal cortex, known to be involved in suppression of prepotent response tendencies and inhibitory cognitive control. Overall, these findings are indicative of an age-related shift along the continuum from automatic to more controlled processing of movement. This increased cognitive monitoring of movement refers to enhanced attentional deployment, more pronounced processing of sensory information, and intersensory integration.

New methodology in infant operant kicking procedures: computerized stimulus control and computerized measurement of kicking

Numerous developmental issues have been assessed using a mobile reinforcement procedure (leg kicks=operant response, movement of a 3D mobile=reinforcer). The current study addressed issues of stimulus control in the mobile procedure through implementation of computerized measurement of leg kicks and computerized control over auditory/visual reinforcers. Experiment 1 demonstrated that the computerized apparatus provides a sensitive measure of learning and memory. Experiment 2 examined the efficacy of single and multiple modality reinforcement in learning/retention of operant kicking and revealed successful acquisition with single modality reinforcement. These results suggest that the mobile procedure could be expanded to address other types of developmental questions (i.e. perception, cognition, intersensory integration) and could be modified to allow replacement of the 3D mobile with images displayed on a computer monitor, thereby extending the utility of the procedure to studies requiring precise psychophysical control over visual stimuli (i.e. object identification and categorization).

Digit Span Changes From Puberty to Old Age Under Different Levels of Education

The goal of this study is to demonstrate the age-related changes in multimodality digit span under a research design in which level of education is controlled. Volunteer participants (n 5 1183) were distributed over levels of age (13-98 years) and education (5-8, 9-11, and 121 years). Digit span was measured through 11 scores of the Visual Aural Digit Span Test-Revised on aural or visual stimulation and oral or written response execution, thus allowing for the measurement of intra- and intersensory integration. The increase in digit span scores reversed to a decrease with early adulthood. The slope of the regression line was small but significant. A 4 3 3 3 2 multivariate analysis of variance showed a significant effect of age and education on a combined score comprising the 11 digit span scores. Differences of age and education were predicted by the auditory and visual input scores. The article discusses the cognitive correlates and the age-related changes in digit span from the biological standpoint.

Motor Control Alteration in Posturography in Learning-Disabled Children

Background Some authors have mentioned that intersensory integration is damaged in children with learning disabilities (LDs), and other investigators point to motor control alterations in the same patients. Thus, we decided to study these hypotheses by means of posturographic recordings. Methods A highly selected group of 27 children with LDs was compared with 27 children of control group without LDs. Patients and controls were placed on the Equitest equipment platform. Sensory organization tests evaluated different test conditions that systematically vary visual, vestibular, and foot somatosensory cues available to subjects while they attempt to maintain a stable, quiet stance. Movement coordination test involved sudden posterior and anterior translations of the patient support surface. Results No significant correlations between scholastics and posturographic performance were observed. No difference in the six conditions and in sensory organization ratios or in visual preference between both groups was disclosed. Motor control test on children with LDs showed significant higher values in latencies in averages of large translations. Conclusions Our data suggest that vestibular-visual-somatosensory organization for posture control are not abnormal in children with LDs; instead, motor controls show higher latencies with large translation movements, which suggest abnormal rate and timing precision motor coordination.

Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective pictures.

Multisensory integration is a powerful mechanism for increasing adaptive responses, as illustrated by binding of fear expressed in a face with fear present in a voice. To understand the role of awareness in intersensory integration of affective information we studied multisensory integration under conditions of conscious and nonconscious processing of the visual component of an audiovisual stimulus pair. Auditory-event-related potentials were measured in two patients (GY and DB) who were unable to perceive visual stimuli consciously because of striate cortex damage. To explore the role of conscious vision of audiovisual pairing, we also compared audiovisual integration in either naturalistic pairings (a facial expression paired with an emotional voice) or semantic pairings (an emotional picture paired with the same voice). We studied the hypothesis that semantic pairings, unlike naturalistic pairings, might require mediation by intact visual cortex and possibly by feedback to primary cortex from higher cognitive processes. Our results indicate that presenting incongruent visual affective information together with the voice translates as an amplitude decrease of auditory-event-related potentials. This effect obtains for both naturalistic and semantic pairings in the intact field, but is restricted to the naturalistic pairings in the blind field.

Heterogeneity and heterochrony in the development of intersensory perception

It is now well established that a variety of intersensory perceptual skills emerge in early human development. Empirical evidence from studies in the author’s as well as other laboratories charting the developmental emergence of these abilities is reviewed. The evidence is considered in terms of the currently dominant theoretical view of intersensory development that assigns the detection of amodal invariants a primary and foundational role. It is argued that this view is inadequate because the detection of amodal invariants is only one of three distinct intersensory integration processes. It is noted that the other two processes, namely, intersensory association of modality-specific cues and non-specific effects of stimulation in one modality on responsiveness to stimulation in another modality, are equally important and that the operation of all three and, in particular, the relation between them, must be studied to attain a complete understanding of intersensory perceptual development. It is suggested that the theoretical approach to the development of intersensory perception should be broadened to include all three types of processes and that developmental studies must respect basic facts and principles of development. To this end, a developmental systems approach is proposed that holds that the development of intersensory integration consists of the heterochronous emergence of heterogeneous perceptual skills.

Driving in the future: temporal visuomotor adaptation and generalization.

Rapid and accurate visuomotor coordination requires tight spatial and temporal sensorimotor synchronization. The introduction of a sensorimotor or intersensory misalignment (either spatial or temporal) impairs performance on most tasks. For more than a century, it has been known that a few minutes of exposure to a spatial misalignment can induce a recalibration of sensorimotor spatial relationships, a phenomenon that may be referred to as spatial visuomotor adaptation. Here, we use a high-fidelity driving simulator to demonstrate that the sensorimotor system can adapt to temporal misalignments on very complex tasks, a phenomenon that we refer to as temporal visuomotor adaptation. We demonstrate that adapting on a single street produces an adaptive state that generalizes to other streets. This shows that temporal visuomotor adaptation is not specific to a single visuomotor transformation, but generalizes across a class of transformations. Temporal visuomotor adaptation is strikingly parallel to spatial visuomotor adaptation, and has strong implications for the understanding of visuomotor coordination and intersensory integration.

Audio-visual integration in schizophrenia

Integration of information provided simultaneously by audition and vision was studied in a group of 18 schizophrenic patients. They were compared to a control group, consisting of 12 normal adults of comparable age and education. By administering two tasks, each focusing on one aspect of audio-visual integration, the study could differentiate between a spatial integration deficit and a speech-based integration deficit. Experiment 1 studied audio-visual interactions in the spatial localisation of sounds. Experiment 2 investigated integration of auditory and visual speech. The schizophrenic group performed as the control group on the sound localisation task, but in the audio-visual speech task, there was an impairment in lipreading as well as a smaller impact of lipreading on auditory speech information. Combined with findings about functional and neuro-anatomical specificity of intersensory integration, the data suggest that there is an integration deficit in the schizophrenic group that is related to the processing of phonetic information.

Situating situated multimodal perception: The relevance of global arrays to development

Stoffregen & Bardy reject the likelihood that infants are sensitive to the global array, implying that intersensory integration is not possible in early development. We argue that infants are sensitive to unimodal arrays and are able to integrate them through the active participation of their nervous system and that the observed developmental changes are due to experience and brain development.

A two-stage model for visual-auditory interaction in saccadic latencies.

In two experiments, saccadic response time (SRT) for eye movements toward visual target stimuli at different horizontal positions was measured under simultaneous or near-simultaneous presentation of an auditory nontarget (distractor). The horizontal position of the auditory signal was varied, using a virtual auditory environment setup. Mean SRT to a visual target increased with distance to the auditory nontarget and with delay of the onset of the auditory signal relative to the onset of the visual stimulus. A stochastic model is presented that distinguishes a peripheral processing stage with separate parallel activation by visual and auditory information from a central processing stage at which intersensory integration takes place. Two model versions differing with respect to the role of the auditory distractors are tested against the SRT data.

Rules of Intersensory Integration in Spatial Scene Analysis and Speechreading

Audiovisual interactions observed in the ventriloquism and McGurk illusions were compared, using the same materials. The ventriloquism effect was estimated through a discordance detection task and the McGurk illusion through an identification task. The stimuli were visually displayed on a screen located in front of the participants’ head and acoustically delivered through one of nine hidden loudspeakers located from straight ahead (0°) to 80° (20° steps) left and right. The speaker’s face was either upright or inverted. The ventriloquism effect was affected by the degree of spatial separation, but unaffected by upright vs. inverted presentation of the face, or by the congruency of the stimuli. The McGurk illusion was of the same size whatever the loudspeaker location but was reduced by face inversion. The differences in the spatial and cognitive rules that govern both interactions are discussed in terms of specific functionality of the underlying mechanisms.