Ventriloquism Effect Research Articles

How bodily perception parallels distal perception

Abstract Bodily perception is standardly contrasted with distal perception. In this paper, I argue that the standard view overlooks a significant respect in which bodily perception parallels distal perception. The parallel is motivated by a comparison between two cross-modal illusions. The ventriloquism effect manifests cross-modal relations between the distal senses of vision and audition and multi-sensory experiences jointly constituted by vision and audition. Thermal referral is a laboratory induced illusion that involves the bodily sensory systems underlying the perception of pressure and heat. I argue that thermal referral indicates the presence of cross-modal relations and multi-sensory experiences of a type sufficiently similar to those manifested by the ventriloquism effect to suggest that pressure and thermal perception parallel vision and audition at the level of bodily perception.

Both stimulus-specific and configurational features of multiple visual stimuli shape the spatial ventriloquism effect.

Studies on multisensory perception often focus on simplistic conditions in which one single stimulus is presented per modality. Yet, in everyday life, we usually encounter multiple signals per modality. To understand how multiple signals within and across the senses are combined, we extended the classical audio-visual spatial ventriloquism paradigm to combine two visual stimuli with one sound. The individual visual stimuli presented in the same trial differed in their relative timing and spatial offsets to the sound, allowing us to contrast their individual and combined influence on sound localization judgements. We find that the ventriloquism bias is not dominated by a single visual stimulus but rather is shaped by the collective multisensory evidence. In particular, the contribution of an individual visual stimulus to the ventriloquism bias depends not only on its own relative spatio-temporal alignment to the sound but also the spatio-temporal alignment of the other visual stimulus. We propose that this pattern of multi-stimulus multisensory integration reflects the evolution of evidence for sensory causal relations during individual trials, calling for the need to extend established models of multisensory causal inference to more naturalistic conditions. Our data also suggest that this pattern of multisensory interactions extends to the ventriloquism aftereffect, a bias in sound localization observed in unisensory judgements following a multisensory stimulus.

Grimes’s Music Video “Shinigami Eyes”

This article draws on recent findings in neuroscience to provide a close reading of Grimes’s music video “Shinigami Eyes.” It includes interviews with a neuroscientist who specializes in multisensory integration, a vision scientist, a color grader, and a media scholar. While multisensory integration is a burgeoning field, accessible texts and reviews are lacking. There haven’t been analyses that apply the findings from the field of multisensory integration to real media objects. Drawing on neuroscience studies, this article explores the concept of Bayesian predictions (a viewer’s experience of a present event weighted against previous experiences); the ways neuroscience explains color, size and placement, and emotion work; and processes such as the inverse effectiveness (when neurons boost both lower-res and higher-res signals, with the most perceptible often for sound), super-additive multisensory effects, the attentional blink, the ventriloquist effect, congruence and incongruity, memory, distractors, sound before image and vice versa, and other features. A conversation with the color grader for “Shinigami Eyes” helps fill things in. This article emerges from a belief that modules on neuroscience and industry studies can be incorporated as components of media study courses, and that neuroscience-informed analyses can facilitate interest and engagement with close readings.

Investigation into Ventriloquism Effect Considering Indication of the Loudspeaker Position

In this work, we investigated strength variations of the ventriloquism effect when the loudspeaker's position was indicated to the subject. The ventriloquism effect is an audiovisual cross‐modality in which a sound is heard from the position of an image, even though the sound and image are presented at different locations. Previously, this effect has been studied under conditions where the sound location is not revealed. In contrast, this paper proposes measuring the ventriloquism effect using a single loudspeaker displayed as a 3D scanned image in the background of a virtual reality (VR) image. The angular discrimination threshold (the perceived angle of disparity between the sound and the image shifted horizontally) was employed to evaluate the ventriloquism effect. The resulting ventriloquism effect was compared to the effect noted in a previous study, where measurements were conducted without revealing the loudspeaker's position to the subjects. The results indicate that revealing the loudspeaker's position emphasizes the ventriloquism effect. © 2023 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Eye responses reflect spatial congruency and perceptual bias in audiovisual interaction

Recent studies have reported that increased pupil dilation and reduced microsaccades are associated with perceptual task enhancement in the bisensory over unisensory conditions. However, little is known about the impacts of bimodal stimulus congruency on those eye responses. We adopted the spatial ventriloquist paradigm to investigate whether spatial congruency of audiovisual stimuli and differences in perceptual experience influence eye responses accompanied by behavioral responses. A simple beep sound and a single disk were used as auditory and visual stimuli respectively. We manipulated spatial congruency of audiovisual stimuli by presenting a visual stimulus on the left or right side of a monitor with an auditory stimulus from the left, right, or center of the monitor. 15 participants performed a sound localization task by clicking their perceived location of the auditory stimulus on the monitor. Participants maintained fixation on a small dot at the center of the monitor with their heads fixed on a chin rest, while their eye responses were recorded using an eye tracker (Eyelink-1000, SR Research). To examine differential perceptual experience in the audiovisual congruence conditions, we classified all responses of audiovisual trials into two groups based on whether the perceived location of the auditory stimulus biased toward the visual stimulus. We found greater pupil dilation accompanied by a greater ventriloquist effect (VE) in the spatially incongruent condition compared to the congruent condition. In addition, a decrease in the number of microsaccades was associated with the auditory perceptual bias towards visual stimuli in the spatially incongruent condition. Our results showed that both spatial congruency and perceptual bias modulate the VE. Results from eye responses suggest that pupillometry is a reliable marker for spatial congruency whereas microsaccades reflect the perceptual bias.

Asymmetric visual capture of virtual sound sources in the distance dimension.

Visual capture describes the tendency of a sound to be mislocalized to the location of a plausible visual target. This effect, also known as the ventriloquist effect, has been extensively studied in humans, but primarily for mismatches in the angular direction between auditory and visual targets. Here, visual capture was examined in the distance dimension using a single visual target (an un-energized loudspeaker) and invisible virtual sound sources presented over headphones. The sound sources were synthesized from binaural impulse-response measurements at distances ranging from 1 to 5 m (0.25 m steps) in the semi-reverberant room (7.7 × 4.2 × 2.7 m3) in which the experiment was conducted. Listeners (n = 11) were asked whether or not the auditory target appeared to be at the same distance as the visual target. Within a block of trials, the visual target was placed at a fixed distance of 1.5, 3, or 4.5 m, and the auditory target varied randomly from trial-to-trial over the sample of measurement distances. The resulting psychometric functions were generally consistent with visual capture in distance, but the capture was asymmetric: Sound sources behind the visual target were more strongly captured than sources in front of the visual target. This asymmetry is consistent with previous reports in the literature, and is shown here to be well predicted by a simple model of sensory integration and decision in which perceived auditory space is compressed logarithmically in distance and has lower resolution than perceived visual space.

Attentional modulations of audiovisual interactions in apparent motion: Temporal ventriloquism effects on perceived visual speed.

The timing of brief stationary sounds has been shown to alter different aspects of visual motion, such as speed estimation. These effects of auditory timing have been explained by temporal ventriloquism and auditory dominance over visual information in the temporal domain. Although previous studies provide unprecedented evidence for the multisensory nature of speed estimation, how attention is involved in these audiovisual interactions remains unclear. Here, we aimed to understand the effects of spatial attention on these audiovisual interactions in time. We utilized a set of audiovisual stimuli that elicit temporal ventriloquism in visual apparent motion and asked participants to perform a speed comparison task. We manipulated attention either in the visual or auditory domain and systematically changed the number of moving objects in the visual field. When attention was diverted to a stationary object in the visual field via a secondary task, the temporal ventriloquism effects on perceived speed decreased. On the other hand, focusing attention on the auditory stimuli facilitated these effects consistently across different difficulty levels of secondary auditory task. Moreover, the effects of auditory timing on perceived speed did not change with the number of moving objects and existed in all the experimental conditions. Taken together, our findings revealed differential effects of allocating attentional resources in the visual and auditory domains. These behavioral results also demonstrate that reliable temporal ventriloquism effects on visual motion can be induced even in the presence of multiple moving objects in the visual field and under different perceptual load conditions.

A Dynamic Neural Field Model of Multimodal Merging: Application to the Ventriloquist Effect.

Multimodal merging encompasses the ability to localize stimuli based on imprecise information sampled through individual senses such as sight and hearing. Merging decisions are standardly described using Bayesian models that fit behaviors over many trials, encapsulated in a probability distribution. We introduce a novel computational model based on dynamic neural fields able to simulate decision dynamics and generate localization decisions, trial by trial, adapting to varying degrees of discrepancy between audio and visual stimulations. Neural fields are commonly used to model neural processes at a mesoscopic scale-for instance, neurophysiological activity in the superior colliculus. Our model is fit to human psychophysical data of the ventriloquist effect, additionally testing the influence of retinotopic projection onto the superior colliculus and providing a quantitative performance comparison to the Bayesian reference model. While models perform equally on average, a qualitative analysis of free parameters in our model allows insights into the dynamics of the decision and the individual variations in perception caused by noise. We finally show that the increase in the number of free parameters does not result in overfitting and that the parameter space may be either reduced to fit specific criteria or exploited to perform well on more demanding tasks in the future. Indeed, beyond decision or localization tasks, our model opens the door to the simulation of behavioral dynamics, as well as saccade generation driven by multimodal stimulation.

Mary Price’s Renaissances in John Fuller’s Work

This article looks at the various forms of renaissance related to Mary Price, John Fuller’s enigmatic Welsh muse. This ungraspable peasant has indeed haunted the contemporary British writer’s poetry and prose for more than forty years. Not only does he resort to a number of ploys like metonymy or prosopopeia to stage her rebirths, but he also revives classical forms and genres like the pastoral, the elegy or the ekphrasis while challenging their codes. Nevertheless, this gives rise to flawed lacunary portraits giving little information about Mary Price who becomes a spectre of herself. Fuller’s poetics of renaissance is uncannily based on a sense of loss and failure highlighting and strengthening other connections between the living and the dead, men and matter. The more the poet endeavours to capture his muse, the more she escapes him—even when he fictionalises her—, until she imposes herself as his double. Whenever Mary Price is given a voice, the ventriloquism effects are all the more forceful as they evidence an ecopoetic approach to the world replacing man at the heart of natural cycles of death and rebirth characterising his habitat.

Audiovisual spatial recalibration but not integration is shaped by early sensory experience

SummaryTo clarify the role of sensory experience during early development for adult multisensory learning capabilities, we probed audiovisual spatial processing in human individuals who had been born blind because of dense congenital cataracts (CCs) and who subsequently had received cataract removal surgery, some not before adolescence or adulthood. Their ability to integrate audio-visual input and to recalibrate multisensory spatial representations was compared to normally sighted control participants and individuals with a history of developmental (later onset) cataracts. Results in CC individuals revealed both normal multisensory integration in audiovisual trials (ventriloquism effect) and normal recalibration of unimodal auditory localization following audiovisual discrepant exposure (ventriloquism aftereffect) as observed in the control groups. In addition, only the CC group recalibrated unimodal visual localization after audiovisual exposure. Thus, in parallel to typical multisensory integration and learning, atypical crossmodal mechanisms coexisted in CC individuals, suggesting that multisensory recalibration capabilities are defined during a sensitive period in development.

Direct eye gaze enhances the ventriloquism effect

The “ventriloquism effect” describes an illusory phenomenon where the perceived location of an auditory stimulus is pulled toward the location of a visual stimulus. Ventriloquists use this phenomenon to create an illusion where an inanimate puppet is perceived to speak. Ventriloquists use the expression and suppression of their own and the puppet’s mouth movements as well the direction of their respective eye gaze to maximize the illusion. While the puppet’s often exaggerated mouth movements have been demonstrated to enhance the ventriloquism effect, the contribution of direct eye gaze remains unknown. In Experiment 1, participants viewed an image of a person’s face while hearing a temporally synchronous recording of a voice originating from different locations on the azimuthal plane. The eyes of the facial stimuli were either looking directly at participants or were closed. Participants were more likely to misperceive the location of a range of voice locations as coming from a central position when the eye gaze of the facial stimuli were directed toward them. Thus, direct gaze enhances the ventriloquist effect by attracting participants’ perception of the voice locations toward the location of the face. In an exploratory analysis, we furthermore found no evidence for an other-race effect between White vs Asian listeners. In Experiment 2, we replicated the effect of direct eye gaze on the ventriloquism effect, also showing that faces per se attract perceived sound locations compared with audio-only sound localization. Showing a modulation of the ventriloquism effect by socially-salient eye gaze information thus adds to previous findings reporting top-down influences on this effect.

The Ventriloquist Effect is not Consistently Affected by Stimulus Realism

Despite more than 60 years of research, it has remained uncertain if and how realism affects the ventriloquist effect. Here, a sound localization experiment was run using spatially disparate audio-visual stimuli. The visual stimuli were presented using virtual reality, allowing for easy manipulation of the degree of realism of the stimuli. Starting from stimuli commonly used in ventriloquist experiments, i.e., a light flash and noise burst, a new factor was added or changed in each condition to investigate the effect of movement and realism without confounding the effects of an increased temporal correlation of the audio-visual stimuli. First, a distractor task was introduced to ensure that participants fixated their eye gaze during the experiment. Next, movement was added to the visual stimuli while maintaining a similar temporal correlation between the stimuli. Finally, by changing the stimuli from the flash and noise stimuli to the visuals of a bouncing ball that made a matching impact sound, the effect of realism was assessed. No evidence for an effect of realism and movement of the stimuli was found, suggesting that, in simple scenarios, the ventriloquist effect might not be affected by stimulus realism.

Auditory-visual interactions in egocentric distance perception: Ventriloquism effect and aftereffect.

This study describes data on auditory-visual integration and visually-guided adaptation of auditory distance perception using the ventriloquism effect (VE) and ventriloquism aftereffect (VAE). In an experiment, participants judged egocentric distance of interleaved auditory or auditory-visual stimuli with the auditory component located from 0.7 to 2.04 m in front of listeners in a real reverberant environment. The visual component of auditory-visual stimuli was displaced 30% closer (V-closer), 30% farther (V-farther), or aligned (V-aligned) with respect to the auditory component. The VE and VAE were measured in auditory and auditory-visual trials, respectively. Both effects were approximately independent of target distance when expressed in logarithmic units. The VE strength, defined as a difference of V-misaligned and V-aligned response bias, was approximately 72% of the auditory-visual disparity regardless of the visual-displacement direction, while the VAE was stronger in the V-farther (44%) than the V-closer (31%) condition. The VAE persisted to post-adaptation auditory-only blocks of trials, although it was diminished. The rates of build-up/break-down of the VAE were asymmetrical, with slower adaptation in the V-closer condition. These results suggest that auditory-visual distance integration is independent of the direction of induced shift, while the re-calibration is stronger and faster when evoked by more distant visual stimuli.

The relationship between spatial release from masking and the ventriloquist effect

Speech is often multisensory and the perception of auditory-visual cues may not always match reality. This experiment explored how spatial release from masking (SRM) can be influenced by perceptual shifts in spatial location through the ventriloquist effect, and whether such shifts impact speech perception. Normal hearing participants completed two tasks: a localization task and an intelligibility task. Both tasks employed audio-video recordings of the Coordinate Response Measure [Kishline et al., J. Acoust. Soc. Am. 148, 492–495 (2020)] and participants were asked to indicate the location of a talker or to report keywords spoken by one talker in the presence of a competing talker. Participants were exposed to auditory-only and auditory-visual conditions of each task, and in the auditory-visual conditions, audio location was either matched or mismatched with the video location. Listeners showed strong SRM on keyword identification in auditory only conditions and strong visual capture of the auditory target to the talker irrespective of true auditory location. Despite this combination of results predicting a strong perceived SRM elicited by ventriloquism, only moderate perceived SRM was observed from mismatched auditory-visual conditions. Further discussion will address the relative contribution of visuals in matched conditions and effects of perceptual SRM.

The association between audio-visual spatial integration and hallucinations in Schizophrenia

It has been suggested that patients with schizophrenia (SZs) show impairment in multisensory integration (De Gelder et al., 2003; Stevenson et al., 2017; Williams et al., 2010). A work in our group examined the audio-visual temporal integration with sound-induced flash illusion and found a lengthened temporal binding window for the bisensory information in SZs (Kim et al., IMRF 2017). The present study investigated the audio-visual spatial integration in schizophrenia by using the ventriloquist illusion paradigm. Ventriloquist illusion indicates biased perception of the spatial position of an auditory stimulus toward the position of a temporally synchronized visual stimulus. A group of 22 SZs and a group of 22 healthy controls performed an auditory localization task when a beep sounded with or without a synchronized LED flash. Eight loudspeakers with attached LEDs were arranged in a semicircular array, half of which were on the left, and the other half were on the right side of the central fixation LED. Each loudspeaker was 14-deg away from its adjacent one. The audio-visual stimuli were presented 0-deg, 14-deg or 28-deg apart. The ventriloquist effect was defined as the ratio of biased sound localization towards the visual stimulus to the spatial disparity between audio-visual stimuli. Following the experiment, SZs were given a structured psychiatric interview to assess the severity of positive and negative symptoms with SAPS and SANS respectively. Results showed a typical ventriloquist effect in both groups with no statistically significant difference. However, in SZs, the ventriloquist effect showed a negative correlation with the severity of visual hallucinations, and a marginally significant negative correlation with the global ratings of hallucinations. There was no correlation between hallucinations and unisensory auditory localization performance. These results demonstrate that hallucinations were implicated in reduced audio-visual spatial integration, which might be related to impaired multisensory processing, not unisensory processing.

The Effects of Cue Reliability on Crossmodal Recalibration in Adults and Children.

Reliability-based cue combination is a hallmark of multisensory integration, while the role of cue reliability for crossmodal recalibration is less understood. The present study investigated whether visual cue reliability affects audiovisual recalibration in adults and children. Participants had to localize sounds, which were presented either alone or in combination with a spatially discrepant high- or low-reliability visual stimulus. In a previous study we had shown that the ventriloquist effect (indicating multisensory integration) was overall larger in the children groups and that the shift in sound localization toward the spatially discrepant visual stimulus decreased with visual cue reliability in all groups. The present study replicated the onset of the immediate ventriloquist aftereffect (a shift in unimodal sound localization following a single exposure of a spatially discrepant audiovisual stimulus) at the age of 6-7 years. In adults the immediate ventriloquist aftereffect depended on visual cue reliability, whereas the cumulative ventriloquist aftereffect (reflecting the audiovisual spatial discrepancies over the complete experiment) did not. In 6-7-year-olds the immediate ventriloquist aftereffect was independent of visual cue reliability. The present results are compatible with the idea of immediate and cumulative crossmodal recalibrations being dissociable processes and that the immediate ventriloquist aftereffect is more closely related to genuine multisensory integration.

Semantic and spatial congruency mould audiovisual integration depending on perceptual awareness

Information integration is considered a hallmark of human consciousness. Recent research has challenged this tenet by showing multisensory interactions in the absence of awareness. This psychophysics study assessed the impact of spatial and semantic correspondences on audiovisual binding in the presence and absence of visual awareness by combining forward–backward masking with spatial ventriloquism. Observers were presented with object pictures and synchronous sounds that were spatially and/or semantically congruent or incongruent. On each trial observers located the sound, identified the picture and rated the picture’s visibility. We observed a robust ventriloquist effect for subjectively visible and invisible pictures indicating that pictures that evade our perceptual awareness influence where we perceive sounds. Critically, semantic congruency enhanced these visual biases on perceived sound location only when the picture entered observers’ awareness. Our results demonstrate that crossmodal influences operating from vision to audition and vice versa are interactively controlled by spatial and semantic congruency in the presence of awareness. However, when visual processing is disrupted by masking procedures audiovisual interactions no longer depend on semantic correspondences.

The Ventriloquist Effect is not Consistently Affected by Stimulus Realism †

Despite more than 60 years of research, it has remained uncertain if and how realism affects the ventriloquist effect. Here, a sound localization experiment was run using spatially disparate audio-visual stimuli. The visual stimuli were presented using virtual reality, allowing for easy manipulation of the degree of realism of the stimuli. Starting from stimuli commonly used in ventriloquist experiments, i.e., a light flash and noise burst, a new factor was added or changed in each condition to investigate the effect of movement and realism without confounding the effects of an increased temporal correlation of the audio-visual stimuli. First, a distractor task was introduced to ensure that participants fixated their eye gaze during the experiment. Next, movement was added to the visual stimuli while maintaining a similar temporal correlation between the stimuli. Finally, by changing the stimuli from the flash and noise stimuli to the visuals of a bouncing ball that made a matching impact sound, the effect of realism was assessed. No evidence for an effect of realism and movement of the stimuli was found, suggesting that, in simple scenarios, the ventriloquist effect might not be affected by stimulus realism.

The Ventriloquist Effect is not Consistently Affected by Stimulus Realism

Despite more than 60 years of research, it has remained uncertain if and how realism affects the ventriloquist effect. Here, a sound localization experiment was run using spatially disparate audio-visual stimuli. The visual stimuli were presented using virtual reality, allowing for easy manipulation of the degree of realism of the stimuli. Starting from stimuli commonly used in ventriloquist experiments, i.e., a light flash and noise burst, a new factor was added or changed in each condition to investigate the effect of movement and realism without confounding the effects of an increased temporal correlation of the audio-visual stimuli. First, a distractor task was introduced to ensure that participants fixated their eye gaze during the experiment. Next, movement was added to the visual stimuli while maintaining a similar temporal correlation between the stimuli. Finally, by changing the stimuli from the flash and noise stimuli to the visuals of a bouncing ball that made a matching impact sound, the effect of realism was assessed. No evidence for an effect of realism and movement of the stimuli was found, suggesting that, in simple scenarios, the ventriloquist effect might not be affected by stimulus realism.

Sensory- and memory-related drivers for altered ventriloquism effects and aftereffects in older adults

The manner in which humans exploit multisensory information for subsequent decisions changes with age. Multiple causes for such age-effects are being discussed, including a reduced precision in peripheral sensory representations, changes in cognitive inference about causal relations between sensory cues, and a decline in memory contributing to altered sequential patterns of multisensory behaviour. To dissociate these putative contributions, we investigated how healthy young and older adults integrate audio-visual spatial information within trials (the ventriloquism effect) and between trials (the ventriloquism aftereffect). With both a model-free and (Bayesian) model-based analyses we found that both biases differed between groups. Our results attribute the age-change in the ventriloquism bias to a decline in spatial hearing rather than a change in cognitive processes. This decline in peripheral function, combined with a more prominent influence from preceding responses rather than preceding stimuli in the elderly, can also explain the observed age-effect in the ventriloquism aftereffect. Our results suggest a transition from a sensory-to a behavior-driven influence of past multisensory experience on perceptual decisions with age, due to reduced sensory precision and change in memory capacity.