Audiovisual Synchrony Perception Research Articles

Quantifying Egocentric Distance Perception in Virtual Reality Environment

In virtual reality (VR) studies, where object distance plays a role of an independent variable, unknown egocentric distance perception values can affect the interpretation of the collected data. It is known that the perceived egocentric distance in VR is often underestimated, which may affect other judgments that implicitly depend on it. In order to prepare later experiments on the effect of distance on audiovisual (a)synchrony perception, this study quantifies the egocentric distance perception in a virtual indoor environment using two methods: verbal judgment (VJ) and position adjustment (PA). For the VJ method, participants verbally estimated the distance between their own position and a cardboard box position at a distance between a nominal 5 m and 30 m, with increments of 5 m. For the PA method, participants were asked to position a cardboard box to an instructed distance of a nominal 5 m to 13 m, with increments of 1 m. Both methods (VJ and PA) showed significant and substantial levels of underestimation, where simulated distance was underestimated on average by 38.5%. Our study suggests taking these findings into account when treating distance as an independent parameter in experiments conducted in purely simulated virtual spaces which do not exist physically in the real world.

Perception of audio-visual synchrony in infants at elevated likelihood of developing autism spectrum disorder.

Findings suggest that early deficits in multisensory integration may impair language development among infants with elevated likelihood of developing ASD. •Perceptual integration of auditory and visual cues within speech is important for language development. •Prior work suggests that children with ASD are less sensitive to the temporal synchrony within audio-visual speech. •In this study, infants at elevated likelihood of developing ASD showed a larger temporal binding window for adynamic social event (Speaking Face) than TD infants, suggesting less efficient multisensory integration.

Neural correlates of audiovisual speech synchrony perception and its relationship with autistic traits.

The anterior insula (AI) has the central role in coordinating attention and integrating information from multiple sensory modalities. AI dysfunction may contribute to both sensory and social impairments in autism spectrum disorder (ASD). Little is known regarding the brain mechanisms that guide multisensory integration, and how such neural activity might be affected by autistic-like symptoms in the general population. In this study, 72 healthy young adults performed an audiovisual speech synchrony judgment (SJ) task during fMRI scanning. We aimed to investigate the SJ-related brain activations and connectivity, with a focus on the AI. Compared with synchronous speech, asynchrony perception triggered stronger activations in the bilateral AI, and other frontal-cingulate-parietal regions. In contrast, synchronous perception resulted in greater involvement of the primary auditory and visual areas, indicating multisensory validation and fusion. Moreover, the AI demonstrated a stronger connection with the anterior cingulate gyrus (ACC) in the audiovisual asynchronous (vs. synchronous) condition. To facilitate asynchrony detection, the AI may integrate auditory and visual speech stimuli, and generate a control signal to the ACC that further supports conflict-resolving and response selection. Correlation analysis, however, suggested that audiovisual synchrony perception and its related AI activation and connectivity did not significantly vary with different levels of autistic traits. These findings provide novel evidence for the neural mechanisms underlying multisensory temporal processing in healthy people. Future research should examine whether such findings would be extended to ASD patients.

No differences between adults with and without autism in audiovisual synchrony perception.

It has been known for a long time that individuals diagnosed with autism spectrum disorder perceive the world differently. In this study, we investigated how people with or without autism perceive visual and auditory information. We know that an auditory and a visual stimulus do not have to be perfectly synchronous for us to perceive them as synchronous: first, when the two are within a certain time window (temporal binding window), the brain will tell us that they are synchronous. Second, the brain can also adapt quickly to audiovisual asynchronies (rapid recalibration). Although previous studies have shown that people with autism spectrum disorder have different temporal binding windows, and less rapid recalibration, we did not find these differences in our study. However, we did find that both processes develop over age, and since previous studies tested only young people (children, adolescents, and young adults), and we tested adults from 18 to 55 years, this might explain the different findings. In the end, there might be quite a complex story, where people with and without autism spectrum disorder perceive the world differently, even dependent on how old they are.

Visual field differences in temporal synchrony processing for audio-visual stimuli.

Audio-visual integration relies on temporal synchrony between visual and auditory inputs. However, differences in traveling and transmitting speeds between visual and auditory stimuli exist; therefore, audio-visual synchrony perception exhibits flexible functions. The processing speed of visual stimuli affects the perception of audio-visual synchrony. The present study examined the effects of visual fields, in which visual stimuli are presented, for the processing of audio-visual temporal synchrony. The point of subjective simultaneity, the temporal binding window, and the rapid recalibration effect were measured using temporal order judgment, simultaneity judgment, and stream/bounce perception, because different mechanisms of temporal processing have been suggested among these three paradigms. The results indicate that auditory stimuli should be presented earlier for visual stimuli in the central visual field than in the peripheral visual field condition in order to perceive subjective simultaneity in the temporal order judgment task conducted in this study. Meanwhile, the subjective simultaneity bandwidth was broader in the central visual field than in the peripheral visual field during the simultaneity judgment task. In the stream/bounce perception task, neither the point of subjective simultaneity nor the temporal binding window differed between the two types of visual fields. Moreover, rapid recalibration occurred in both visual fields during the simultaneity judgment tasks. However, during the temporal order judgment task and stream/bounce perception, rapid recalibration occurred only in the central visual field. These results suggest that differences in visual processing speed based on the visual field modulate the temporal processing of audio-visual stimuli. Furthermore, these three tasks, temporal order judgment, simultaneity judgment, and stream/bounce perception, each have distinct functional characteristics for audio-visual synchrony perception. Future studies are necessary to confirm the effects of compensation regarding differences in the temporal resolution of the visual field in later cortical visual pathways on visual field differences in audio-visual temporal synchrony.

Audio-Visual Causality and Stimulus Reliability Affect Audio-Visual Synchrony Perception.

People can discriminate the synchrony between audio-visual scenes. However, the sensitivity of audio-visual synchrony perception can be affected by many factors. Using a simultaneity judgment task, the present study investigated whether the synchrony perception of complex audio-visual stimuli was affected by audio-visual causality and stimulus reliability. In Experiment 1, the results showed that audio-visual causality could increase one's sensitivity to audio-visual onset asynchrony (AVOA) of both action stimuli and speech stimuli. Moreover, participants were more tolerant of AVOA of speech stimuli than that of action stimuli in the high causality condition, whereas no significant difference between these two kinds of stimuli was found in the low causality condition. In Experiment 2, the speech stimuli were manipulated with either high or low stimulus reliability. The results revealed a significant interaction between audio-visual causality and stimulus reliability. Under the low causality condition, the percentage of “synchronous” responses of audio-visual intact stimuli was significantly higher than that of visual_intact/auditory_blurred stimuli and audio-visual blurred stimuli. In contrast, no significant difference among all levels of stimulus reliability was observed under the high causality condition. Our study supported the synergistic effect of top-down processing and bottom-up processing in audio-visual synchrony perception.

Rapid Temporal Recalibration to Audiovisual Asynchrony Occurs Across the Difference in Neural Processing Speed Based on Spatial Frequency.

Audiovisual integration relies on temporal synchrony between visual and auditory stimuli. The brain rapidly adapts to audiovisual asynchronous events by shifting the timing of subjective synchrony in the direction of the leading modality of the most recent event, a process called rapid temporal recalibration. This phenomenon is the flexible function of audiovisual synchrony perception. Previous studies found that neural processing speed based on spatial frequency (SF) affects the timing of subjective synchrony. This study examined the effects of SF on the rapid temporal recalibration process by discriminating whether the presentation of the visual and auditory stimuli was simultaneous. I compared the magnitudes of the recalibration effect between low and high SF visual stimuli using two techniques. First, I randomly presented each SF accompanied by a tone during one session, then in a second experiment, only a single SF was paired with the tone throughout the one session. The results indicated that rapid recalibration occurred regardless of difference in presented SF between preceding and test trials. The recalibration magnitude did not significantly differ between the SF conditions. These findings confirm that intersensory temporal process is important to produce rapid recalibration and suggest that rapid recalibration can be induced by the simultaneity judgment criterion changes attributed to the low-level temporal information of audiovisual events.

Perception of causality and synchrony dissociate in the audiovisual bounce-inducing effect (ABE)

A sound can cause 2 visual streaming objects appear to bounce (the audiovisual bounce-inducing effect, ABE). Here we examined whether the stream/bounce percept affects perception of audiovisual synchrony. Participants saw 2 disks that either clearly streamed, clearly bounced, or were ambiguous, and heard a sound around the point of contact (POC). They reported, on each trial, whether they perceived the disks to ‘stream’ or ‘bounce’, and whether the sound was ‘synchronous’ or ‘asynchronous’ with the POC. Results showed that the optimal time of the sound to induce a bounce was before the POC (−59 msec), whereas audiovisual synchrony was maximal when the sound came after the POC (+16 msec). The range of temporal asynchronies perceived as ‘synchronous’, the temporal binding window (TBW), was wider when disks were perceived as bouncing than streaming, with no difference between ambiguous and non-ambiguous visual displays. These results demonstrate 1) that causality differs from synchrony, 2) that causality widens the TBW, and 3) that the ABE is perceptually real.

Audiovisual synchrony perception in observing human motion to music.

To examine how individuals perceive synchrony between music and body motion, we investigated the characteristics of synchrony perception during observation of a Japanese Radio Calisthenics routine. We used the constant stimuli method to present video clips of an individual performing an exercise routine. We generated stimuli with a range of temporal shifts between the visual and auditory streams, and asked participants to make synchrony judgments. We then examined which movement-feature points agreed with music beats when the participants perceived synchrony. We found that extremities (e.g., hands and feet) reached the movement endpoint or moved through the lowest position at music beats associated with synchrony. Movement onsets never agreed with music beats. To investigate whether visual information about the feature points was necessary for synchrony perception, we conducted a second experiment where only limited portions of video clips were presented to the participants. Participants consistently judged synchrony even when the video image did not contain the critical feature points, suggesting that a prediction mechanism contributes to synchrony perception. To discuss the meaning of these feature points with respect to synchrony perception, we examined the temporal relationship between the motion of body parts and the ground reaction force (GRF) of exercise performers, which reflected the total force acting on the performer. Interestingly, vertical GRF showed local peaks consistently synchronized with music beats for most exercises, with timing that was closely correlated with the timing of movement feature points. This result suggests that synchrony perception in humans is based on some global variable anticipated from visual information, instead of the feature points found in the motion of individual body parts. In summary, the present results indicate that synchrony perception during observation of human motion to music depends largely on spatiotemporal prediction of the performer’s motion.

Behavioral Plasticity of Audiovisual Perception: Rapid Recalibration of Temporal Sensitivity but Not Perceptual Binding Following Adult-Onset Hearing Loss.

The ability to accurately integrate or bind stimuli from more than one sensory modality is highly dependent on the features of the stimuli, such as their intensity and relative timing. Previous studies have demonstrated that the ability to perceptually bind stimuli is impaired in various clinical conditions such as autism, dyslexia, schizophrenia, as well as aging. However, it remains unknown if adult-onset hearing loss, separate from aging, influences audiovisual temporal acuity. In the present study, rats were trained using appetitive operant conditioning to perform an audiovisual temporal order judgment (TOJ) task or synchrony judgment (SJ) task in order to investigate the nature and extent that audiovisual temporal acuity is affected by adult-onset hearing loss, with a specific focus on the time-course of perceptual changes following loud noise exposure. In our first series of experiments, we found that audiovisual temporal acuity in normal-hearing rats was influenced by sound intensity, such that when a quieter sound was presented, the rats were biased to perceive the audiovisual stimuli as asynchronous (SJ task), or as though the visual stimulus was presented first (TOJ task). Psychophysical testing demonstrated that noise-induced hearing loss did not alter the rats’ temporal sensitivity 2–3 weeks post-noise exposure, despite rats showing an initial difficulty in differentiating the temporal order of audiovisual stimuli. Furthermore, consistent with normal-hearing rats, the timing at which the stimuli were perceived as simultaneous (i.e., the point of subjective simultaneity, PSS) remained sensitive to sound intensity following hearing loss. Contrary to the TOJ task, hearing loss resulted in persistent impairments in asynchrony detection during the SJ task, such that a greater proportion of trials were now perceived as synchronous. Moreover, psychophysical testing found that noise-exposed rats had altered audiovisual synchrony perception, consistent with impaired audiovisual perceptual binding (e.g., an increase in the temporal window of integration on the right side of simultaneity; right temporal binding window (TBW)). Ultimately, our collective results show for the first time that adult-onset hearing loss leads to behavioral plasticity of audiovisual perception, characterized by a rapid recalibration of temporal sensitivity but a persistent impairment in the perceptual binding of audiovisual stimuli.

Does native language temporal experience transfer to audio-visual synchrony perception?

The temporal alignment of what we hear and see is fundamental for the cognitive organization of information from our environment. Research indicates that a perceiver's experience influences sensitivity to audio-visual (AV) synchrony. We theorize that experience that enhances sensitivity to speech sound distinctions in the temporal domain would enhance sensitivity in AV synchrony perception. With this basis, a perceiver whose native language (L1) involves duration-based phonemic distinctions would be expected to be more sensitive to AV synchrony in speech than for an L1 which has less use of temporal cues. In the current study, simultaneity judgment data for the syllable /ba/ were collected with 23 steps of AV alignments: from audio preceding the video (audio-lead) to the audio and video being physically aligned (synchronous) to video preceding the audio (video-lead). Two groups of participants differing in L1 experience with phonemic duration were included: native speakers of Norwegian (binary phonemic quantity distinction) and English (no phonemic quantity distinction). Preliminary results based on measures the audio-lead threshold (ALT) support the hypothesis that native language experience may influence broad mechanisms of timing, such as those moderating AV synchrony perception. Findings contribute to understanding the underpinnings of experience and AV synchrony perception.

Effects of Temporal Distribution on Utility of TemporalFactors in Competitive Audio-Visual PerceivedSynchrony.

The perception of audio-visual synchrony is affected by both temporal coincidence and stimulus congruency factors. In situations when temporal and stimulus information are not in agreement, the perceiver must rely on the relative informative value of both factors in deciding which of multiple potential binding candidates are most likely to be of a common source to a target. Previous research has shown that, all being equal, participants tend to rely primarily on temporal information, and only take stimulus information into consideration when temporal information is ambiguous. The current research seeks to examine the reliance on temporal vs. stimulus information by altering the degree of useful information available in temporal aspects. By varying the temporal distribution of stimuli, it was possible to either increase or decrease the number of trials on which temporal information is conclusive. Data indicate that when temporal information is less informative (i.e., when more asynchronous stimuli are presented), we become less reliant on using prior knowledge about timing relationships when making synchrony judgements. However, when temporal information is more informative (i.e., when more synchronous stimuli are presented) there is no increase in reliance on this type of information. These findings increase what is known about competitive audiovisual processing, and the fact that temporal information serves as a kind of default stimulus property, which can be decreased by reducing the utility of that information.

Babies know bad dancing when they see it: Older but not younger infants discriminate between synchronous and asynchronous audiovisual musical displays

Movement to music is a universal human behavior, yet little is known about how observers perceive audiovisual synchrony in complex musical displays such as a person dancing to music, particularly during infancy and childhood. In the current study, we investigated how perception of musical audiovisual synchrony develops over the first year of life. We habituated infants to a video of a person dancing to music and subsequently presented videos in which the visual track was matched (synchronous) or mismatched (asynchronous) with the audio track. In a visual-only control condition, we presented the same visual stimuli with no sound. In Experiment 1, we found that older infants (8–12months) exhibited a novelty preference for the mismatched movie when both auditory information and visual information were available and showed no preference when only visual information was available. By contrast, younger infants (5–8months) in Experiment 2 did not discriminate matching stimuli from mismatching stimuli. This suggests that the ability to perceive musical audiovisual synchrony may develop during the second half of the first year of infancy.

Finger-to-Beat Coordination Skill of Non-dancers, Street Dancers, and the World Champion of a Street-Dance Competition.

The coordination of body movements to a musical beat is a common feature of many dance styles. However, the auditory–motor coordination skills of dancers remain largely uninvestigated. The purpose of this study was to examine the auditory–motor coordination skills of non-dancers, street dancers, and the winner of a celebrated international street dance competition, while coordinating their rhythmic finger movements to a beat. The beat rate of a metronome increased from 1.0 to 3.7 Hz. The participants were asked to either flex or extend their index fingers on the beat in each condition. Under the extend-on-the-beat condition, both the dancers and non-dancers showed a spontaneous transition from the extend-on-the-beat to the flex-on-the-beat or to a phase wandering pattern. However, the critical frequency at which the transition occurred was significantly higher in the dancers (3.3 Hz) than in the non-dancers (2.6 Hz). Under the flex-on-the-beat condition, the dancers were able to maintain their coordination pattern more stably at high beat rates compared to the non-dancers. Furthermore, the world champion matched the timing of movement peak velocity to the beat across the different beat rates. This may give a sense of unity between the movement and the beat for the audience because the peak velocity of the rhythmic movement works as a temporal cue for the audiovisual synchrony perception. These results suggest that the skills of accomplished dancers lie in their small finger movements and that the sensorimotor learning of street dance is characterized by a stabilization of the coordination patterns, including the inhibition of an unintentional transition to other coordination patterns.

Hysteresis in Audiovisual Synchrony Perception

The effect of stimulation history on the perception of a current event can yield two opposite effects, namely: adaptation or hysteresis. The perception of the current event thus goes in the opposite or in the same direction as prior stimulation, respectively. In audiovisual (AV) synchrony perception, adaptation effects have primarily been reported. Here, we tested if perceptual hysteresis could also be observed over adaptation in AV timing perception by varying different experimental conditions. Participants were asked to judge the synchrony of the last (test) stimulus of an AV sequence with either constant or gradually changing AV intervals (constant and dynamic condition, respectively). The onset timing of the test stimulus could be cued or not (prospective vs. retrospective condition, respectively). We observed hysteretic effects for AV synchrony judgments in the retrospective condition that were independent of the constant or dynamic nature of the adapted stimuli; these effects disappeared in the prospective condition. The present findings suggest that knowing when to estimate a stimulus property has a crucial impact on perceptual simultaneity judgments. Our results extend beyond AV timing perception, and have strong implications regarding the comparative study of hysteresis and adaptation phenomena.

Older age results in difficulties separating auditory and visual signals in time.

Previous research provides conflicting evidence regarding whether older adults have altered tolerance to timing differences between auditory and visual events. We examine the potential impact of age-related unisensory decline on audiovisual synchrony perception. Fifteen younger (21-32 years) and 13 older (60-72 years) adults participated. To assess unisensory sensitivity, visual Gabor contrast detection thresholds and auditory masked tone pip detection thresholds were measured. Four multisensory conditions were then tested: suprathreshold and near-threshold stimuli (based on individual unisensory psychometric functions), each tested with a masked tone pip stimuli at 0.5 and 4 kHz sound frequencies. Two audiovisual pairs (one synchronous, the other asynchronous) were presented in a two-interval forced-choice procedure, with observers identifying the interval containing the asynchronous stimulus. Older adults required a larger physical asynchrony to perceive the stimuli as asynchronous, particularly for low frequency sounds. Our results demonstrate that the impact of age on audiovisual synchrony perception cannot be explained by decline in unisensory sensitivity alone.

Reduced audiovisual recalibration in the elderly

Perceived synchrony of visual and auditory signals can be altered by exposure to a stream of temporally offset stimulus pairs. Previous literature suggests that adapting to audiovisual temporal offsets is an important recalibration to correctly combine audiovisual stimuli into a single percept across a range of source distances. Healthy aging results in synchrony perception over a wider range of temporally offset visual and auditory signals, independent of age-related unisensory declines in vision and hearing sensitivities. However, the impact of aging on audiovisual recalibration is unknown. Audiovisual synchrony perception for sound-lead and sound-lag stimuli was measured for 15 younger (22–32 years old) and 15 older (64–74 years old) healthy adults using a method-of-constant-stimuli, after adapting to a stream of visual and auditory pairs. The adaptation pairs were either synchronous or asynchronous (sound-lag of 230 ms). The adaptation effect for each observer was computed as the shift in the mean of the individually fitted psychometric functions after adapting to asynchrony. Post-adaptation to synchrony, the younger and older observers had average window widths (±standard deviation) of 326 (±80) and 448 (±105) ms, respectively. There was no adaptation effect for sound-lead pairs. Both the younger and older observers, however, perceived more sound-lag pairs as synchronous. The magnitude of the adaptation effect in the older observers was not correlated with how often they saw the adapting sound-lag stimuli as asynchronous. Our finding demonstrates that audiovisual synchrony perception adapts less with advancing age.

Reduced audiovisual recalibration in the elderly

Perceived synchrony of visual and auditory signals can be altered by exposure to a stream of temporally offset stimulus pairs. Previous literature suggests that adapting to audiovisual temporal offsets is an important recalibration to correctly combine audiovisual stimuli into a single percept across a range of source distances. Healthy ageing results in synchrony perception over a wider range of temporally offset visual and auditory signals, independent of age-related unisensory declines in vision and hearing sensitivities. However the impact of ageing on audiovisual recalibration is unkonwn. Audiovisual synchrony perception for sound-lead and sound-lag stimuli was measured for fifteen younger (22-32 years old) and fifteen older (64-74 years old) healthy adults using a method-of-constant-stimuli, after adapting to a stream of visual and auditory pairs. The adaptation pairs were either synchronous or asynchronous (sound-lag of 230ms). The adaptation effect for each observer was computed as the shift in the mean of the individually fitted psychometric functions after adapting to asynchrony. Post adaptation to synchrony, the younger and older observers had average window widths (±standard deviation) of 326 (±80) and 448 (±105) ms, respectively. There was no adaptation effect for sound-lead pairs. Both the younger and older observers however perceived more sound-lag pairs as synchronous. The magnitude of the adaptation effect in the older observers was not correlated with how often they saw the adapting sound-lag stimuli as asynchronous nor their synchrony window widths. Our finding demonstrates that audiovisual synchrony perception adapts less with advancing age.

Window of audio-visual simultaneity is unaffected by spatio-temporal visual clutter

In the present study we investigate the rules governing the perception of audiovisual synchrony within spatio-temporally cluttered visual environments. Participants viewed a ring of 19 discs modulating in luminance while hearing an amplitude modulating tone. Each disc modulated with a unique temporal phase (40 ms intervals), with only one synchronized to the tone. Participants searched for the synchronised disc whose spatial location varied randomly across trials. Square-wave modulation facilitated search: the synchronized disc was frequently chosen, with tight response distributions centred near zero-phase lag. In the sinusoidal condition responses were equally distributed over the 19 discs regardless of phase. To investigate whether subjective synchrony in the square-wave condition was limited by spatial or temporal factors we repeated the experiment with either reduced spatial density (9 discs) or temporal density (80 ms phase intervals). Reduced temporal density greatly facilitated synchrony perception but left the synchrony bandwidth unchanged, while no influence of spatial density was found. We conclude that audio-visual synchrony is not strongly constrained by the spatial or temporal density of the visual display, but by a temporal window within which audio-visual events are perceived as synchronous, with a full bandwidth of ~185 ms.

Peak velocity as a cue in audiovisual synchrony perception of rhythmic stimuli

This study investigated audiovisual synchrony perception in a rhythmic context, where the sound was not consequent upon the observed movement. Participants judged synchrony between a bouncing point-light figure and an auditory rhythm in two experiments. Two questions were of interest: (1) whether the reference in the visual movement, with which the auditory beat should coincide, relies on a position or a velocity cue; (2) whether the figure form and motion profile affect synchrony perception. Experiment 1 required synchrony judgment with regard to the same (lowest) position of the movement in four visual conditions: two figure forms (human or non-human) combined with two motion profiles (human or ball trajectory). Whereas figure form did not affect synchrony perception, the point of subjective simultaneity differed between the two motions, suggesting that participants adopted the peak velocity in each downward trajectory as their visual reference. Experiment 2 further demonstrated that, when judgment was required with regard to the highest position, the maximal synchrony response was considerably low for ball motion, which lacked a peak velocity in the upward trajectory. The finding of peak velocity as a cue parallels results of visuomotor synchronization tasks employing biological stimuli, suggesting that synchrony judgment with rhythmic motions relies on the perceived visual beat.