Audiovisual Integration Effect Research Articles

Impaired emotional multimodal integration in inhibition of return in children with attention-deficit/hyperactivity disorder.

Attention-deficit/hyperactivity disorder (ADHD) is a common neurodevelopmental disorder in children. Previous studies have shown that children with ADHD have impaired processing of emotional stimuli, but it is unclear whether their ability to integrate multimodal emotional stimuli is impaired and at which processing pathway this impairment exists. The present study investigated the ability of children with ADHD to integrate emotional audiovisual stimuli under different emotional conditions, and the effect of audiovisual integration on IOR to reveal the impaired processing pathway of their emotional audiovisual integration. Fifty-eight school-age children (29 with ADHD and 29 matched typically developing (TD) children) performed an emotional valence discrimination task with a cue-target paradigm. The results showed that children with ADHD did not exhibit audiovisual integration of emotional stimuli in all experimental conditions. In addition, the IOR effect was significantly smaller for audiovisual targets than for visual targets under the negative but not the neutral emotion condition in children with ADHD, whereas this effect was present in all emotion conditions in TD children. These results indicate that the ability to integrate emotional audiovisual information is impaired in children with ADHD and this impairment exists in both bottom-up and top-down pathways. Additionally, although presenting emotional auditory stimuli at the same time as emotional faces reduced IOR both in children with ADHD and TD, the manner of reduction differed. These findings provide new evidence of emotional processing deficits and multimodal integration deficits in children with ADHD, and help provide support for children in educational settings.

The Power of Trial History: How Previous Trial Shapes Audiovisual Integration

Abstract Combining information from visual and auditory modalities to form a unified and coherent perception is known as audiovisual integration. Audiovisual integration is affected by many factors. However, it remains unclear whether the trial history can influence audiovisual integration. We used a target–target paradigm to investigate how the target modality and spatial location of the previous trial affect audiovisual integration under conditions of divided-modalities attention (Experiment 1) and modality-specific selective attention (Experiment 2). In Experiment 1, we found that audiovisual integration was enhanced in the repeat locations compared with switch locations. Audiovisual integration was the largest following the auditory targets compared to following the visual and audiovisual targets. In Experiment 2, where participants were asked to focus only on visual, we found that the audiovisual integration effect was larger in the repeat location trials than switch location trials only when the audiovisual target was presented in the previous trial. The present results provide the first evidence that trial history can have an effect on audiovisual integration. The mechanisms of trial history modulating audiovisual integration are discussed. Future examining of audiovisual integration should carefully manipulate experimental conditions based on the effects of trial history.

Modulation of audiovisual integration in the left and right sides: effects of side and spatial coherency

BackgroundUsing event-related potentials (ERPs), we aimed to investigate audiovisual integration neural mechanisms during a letter identification task in the left and right sides. Unimodal (A,V) and bimodal (AV) stimuli were presented on either side, with ERPs from unimodal (A,V) stimuli on the same side being compared to those from simultaneous bimodal stimuli (AV). Non-zero results of the AV-(A + V) difference waveforms indicated audiovisual integration on the left/right side.ResultsWhen spatially coherent AV stimuli were presented on the right side, two significant ERP components in the integrated differential wave were noted. The N134 and N262, present in the first 300 ms of the AV-(A + V) integration difference wave, indicated significant audiovisual integration effects. However, when these stimuli were presented on the left side, there were no significant integration components. This audiovisual integration difference may stem from left/right asymmetry of cerebral hemisphere language processing.ConclusionsAudiovisual letter information presented on the right side was easier to integrate, process, and represent. Additionally, only one significant integrative component peaked at 140 ms in the parietal cortex for spatially non-coherent AV stimuli and provided audiovisual multisensory integration, which could be attributed to some integrative neural processes that depend on the spatial congruity of the auditory and visual stimuli.

Aging effect of cross-modal interactions during audiovisual detection and discrimination by behavior and ERPs.

Numerous studies have shown that aging greatly affects audiovisual integration; however, it is still unclear when the aging effect occurs, and its neural mechanism has yet to be fully elucidated. We assessed the audiovisual integration (AVI) of older (n = 40) and younger (n = 45) adults using simple meaningless stimulus detection and discrimination tasks. The results showed that the response was significantly faster and more accurate for younger adults than for older adults in both the detection and discrimination tasks. The AVI was comparable for older and younger adults during stimulus detection (9.37% vs. 9.43%); however, the AVI was lower for older than for younger adults during stimulus discrimination (9.48% vs. 13.08%) behaviorally. The electroencephalography (EEG) analysis showed that comparable AVI amplitude was found at 220-240 ms for both groups during stimulus detection and discrimination, but there was no significant difference between brain regions for older adults but a higher AVI amplitude in the right posterior for younger adults. Additionally, a significant AVI was found for younger adults in 290-310 ms but was absent for older adults during stimulus discrimination. Furthermore, significant AVI was found in the left anterior and right anterior at 290-310 ms for older adults but in the central, right posterior and left posterior for younger adults. These results suggested that the aging effect of AVI occurred in multiple stages, but the attenuated AVI mainly occurred in the later discriminating stage attributed to attention deficit.

The beep-speed illusion: Non-spatial tones increase perceived speed of visual objects in a forced-choice paradigm

We introduce a new audio-visual illusion revealing the interplay between audio-visual integration and selective visual attention. This illusion involves two simultaneously moving objects that change their motion trajectory occasionally, but only the direction changes of one object are accompanied by spatially uninformative tones. We observed a selective increase in perceived object speed of the audio-visually synchronized object by measuring the point of subjective equality in a forced-choice paradigm. The illusory increase in perceived speed of the audio-visually synchronized object persisted when preventing eye movements. Using temporally matched color changes of the synchronized object also increased the perceived speed. Yet, using color changes of a surrounding frame instead of tones had no effect on perceived speed ruling out simple alertness explanations. Thus, in contrast to coinciding tones, visual coincidences only elicit illusory increases in perceived speed when the coincidence provided spatial information. Taken together, our pattern of results suggests that audio-visual synchrony attracts visual attention toward the coinciding visual object, leading to an increase in speed-perception and thus shedding new light on the interplay between attention and multisensory feature integration. We discuss potential limitations such as the choice of paradigm and outline prospective research question to further investigate the effect of audio-visual integration on perceived object speed.

Developmental Trajectories of Letter and Speech Sound Integration During Reading Acquisition.

Reading acquisition in alphabetic languages starts with learning the associations between speech sounds and letters. This learning process is related to crucial developmental changes of brain regions that serve visual, auditory, multisensory integration, and higher cognitive processes. Here, we studied the development of audiovisual processing and integration of letter-speech sound pairs with an audiovisual target detection functional MRI paradigm. Using a longitudinal approach, we tested children with varying reading outcomes before the start of reading acquisition (T1, 6.5 yo), in first grade (T2, 7.5 yo), and in second grade (T3, 8.5 yo). Early audiovisual integration effects were characterized by higher activation for incongruent than congruent letter-speech sound pairs in the inferior frontal gyrus and ventral occipitotemporal cortex. Audiovisual processing in the left superior temporal gyrus significantly increased from the prereading (T1) to early reading stages (T2, T3). Region of interest analyses revealed that activation in left superior temporal gyrus (STG), inferior frontal gyrus and ventral occipitotemporal cortex increased in children with typical reading fluency skills, while poor readers did not show the same development in these regions. The incongruency effect bilaterally in parts of the STG and insular cortex at T1 was significantly associated with reading fluency skills at T3. These findings provide new insights into the development of the brain circuitry involved in audiovisual processing of letters, the building blocks of words, and reveal early markers of audiovisual integration that may be predictive of reading outcomes.

Inhibition of Return Decreases Early Audiovisual Integration: An Event-Related Potential Study.

Previous behavioral studies have found that inhibition of return decreases the audiovisual integration, while the underlying neural mechanisms are unknown. The current work utilized the high temporal resolution of event-related potentials (ERPs) to investigate how audiovisual integration would be modulated by inhibition of return. We employed the cue-target paradigm and manipulated the target type and cue validity. Participants were required to perform the task of detection of visual (V), auditory (A), or audiovisual (AV) targets shown in the identical (valid cue) or opposed (invalid cue) side to be the preceding exogenous cue. The neural activities between AV targets and the sum of the A and V targets were compared, and their differences were calculated to present the audiovisual integration effect in different cue validity conditions (valid, invalid). The ERPs results showed that a significant super-additive audiovisual integration effect was observed on the P70 (60∼90 ms, frontal-central) only under the invalid cue condition. The significant audiovisual integration effects were observed on the N1 or P2 components (N1, 120∼180 ms, frontal-central-parietal; P2, 200∼260 ms, frontal-central-parietal) in both valid cue as well as invalid cue condition. And there were no significant differences on the later components between invalid cue and valid cue. The result offers the first neural demonstration that inhibition of return modulates the early audiovisual integration process.

Visual Perceptual Load Attenuates Age-Related Audiovisual Integration in an Audiovisual Discrimination Task.

Previous studies confirmed that the cognitive resources are limited for each person, and perceptual load affects the detection of stimulus greatly; however, how the visual perceptual load influences audiovisual integration (AVI) is still unclear. Here, 20 older and 20 younger adults were recruited to perform an auditory/visual discrimination task under various visual perceptual-load conditions. The analysis for the response times revealed a significantly faster response to the audiovisual stimulus than to the visual stimulus or auditory stimulus (all p < 0.001), and a significantly slower response by the older adults than by the younger adults to all targets (all p ≤ 0.024). The race-model analysis revealed a higher AV facilitation effect for older (12.54%) than for younger (7.08%) adults under low visual perceptual-load conditions; however, no obvious difference was found between younger (2.92%) and older (3.06%) adults under medium visual perceptual-load conditions. Only the AV depression effect was found for both younger and older adults under high visual perceptual-load conditions. Additionally, the peak latencies of AVI were significantly delayed in older adults under all visual perceptual-load conditions. These results suggested that visual perceptual load decreased AVI (i.e., depression effects), and the AVI effect was increased but delayed for older adults.

Age-Related Shifts in Theta Oscillatory Activity During Audio-Visual Integration Regardless of Visual Attentional Load.

Audio-visual integration (AVI) is higher in attended conditions than in unattended conditions. Here, we explore the AVI effect when the attentional recourse is competed by additional visual distractors, and its aging effect using single- and dual-tasks. The results showed the highest AVI effect under single-task-attentional-load condition than under no- and dual-task-attentional-load conditions (all P < 0.05) in both older and younger groups, but the AVI effect was weaker and delayed for older adults compared to younger adults for all attentional-load conditions (all P < 0.05). The non-phase-locked oscillation for AVI analysis illustrated the highest theta and alpha oscillatory activity for single-task-attentional-load condition than for no- and dual-task-attentional-load conditions, and the AVI oscillatory activity mainly occurred in the Cz, CP1 and Oz of older adults but in the Fz, FC1, and Cz of younger adults. The AVI effect was significantly negatively correlated with FC1 (r2 = 0.1468, P = 0.05) and Cz (r2 = 0.1447, P = 0.048) theta activity and with Fz (r2 = 0.1557, P = 0.043), FC1 (r2 = 0.1042, P = 0.008), and Cz (r2 = 0.0897, P = 0.010) alpha activity for older adults but not for younger adults in dual task. These results suggested a reduction in AVI ability for peripheral stimuli and a shift in AVI oscillation from anterior to posterior regions in older adults as an adaptive mechanism.

Cue-target onset asynchrony modulates interaction between exogenous attention and audiovisual integration.

Previous studies have shown that exogenous attention decreases audiovisual integration (AVI); however, whether the interaction between exogenous attention and AVI is influenced by cue-target onset asynchrony (CTOA) remains unclear. To clarify this matter, twenty participants were recruited to perform an auditory/visual discrimination task, and they were instructed to respond to the target stimuli as rapidly and accurately as possible. The analysis of the mean response times showed an effective cueing effect under all cued conditions and significant response facilitation for all audiovisual stimuli. A further comparison of the differences between the probability of audiovisual cumulative distributive functions (CDFs) and race model CDFs showed that the AVI latency was shortened under the cued condition relative to that under the no-cue condition, and there was a significant break point when the CTOA was 200ms, with a decrease in the AVI upon going from 100 to 200ms and an increase upon going from 200 to 400ms. These results indicated different mechanisms for the interaction between exogenous attention and the AVI under the shorter and longer CTOA conditions and further suggested that there may be a temporal window in which the AVI effect is mainly affected by exogenous attention, but the interaction might be interfered with by endogenous attention when exceeding the temporal window.

Speech-specific audiovisual integration modulates induced theta-band oscillations.

Speech perception is influenced by vision through a process of audiovisual integration. This is demonstrated by the McGurk illusion where visual speech (for example /ga/) dubbed with incongruent auditory speech (such as /ba/) leads to a modified auditory percept (/da/). Recent studies have indicated that perception of the incongruent speech stimuli used in McGurk paradigms involves mechanisms of both general and audiovisual speech specific mismatch processing and that general mismatch processing modulates induced theta-band (4–8 Hz) oscillations. Here, we investigated whether the theta modulation merely reflects mismatch processing or, alternatively, audiovisual integration of speech. We used electroencephalographic recordings from two previously published studies using audiovisual sine-wave speech (SWS), a spectrally degraded speech signal sounding nonsensical to naïve perceivers but perceived as speech by informed subjects. Earlier studies have shown that informed, but not naïve subjects integrate SWS phonetically with visual speech. In an N1/P2 event-related potential paradigm, we found a significant difference in theta-band activity between informed and naïve perceivers of audiovisual speech, suggesting that audiovisual integration modulates induced theta-band oscillations. In a McGurk mismatch negativity paradigm (MMN) where infrequent McGurk stimuli were embedded in a sequence of frequent audio-visually congruent stimuli we found no difference between congruent and McGurk stimuli. The infrequent stimuli in this paradigm are violating both the general prediction of stimulus content, and that of audiovisual congruence. Hence, we found no support for the hypothesis that audiovisual mismatch modulates induced theta-band oscillations. We also did not find any effects of audiovisual integration in the MMN paradigm, possibly due to the experimental design.

The audio-visual integration effect on music emotion: Behavioral and physiological evidence.

Previous research has indicated that, compared to audio-only presentation, audio-visual congruent presentation can lead to a more intense emotional response. In the present study, we investigated the audio-visual integration effect on emotions elicited by positive or negative music and the role of visual information presentation durations. The participants were presented with audio-only condition, audio-visual congruent condition, and audio-visual incongruent condition and then required to judge the intensity of emotional experience elicited by the music. Their emotional responses to the music were measured using self-ratings and physiological aspects, including heart rate, skin temperature, EMG root mean square and prefrontal EEG. Relative to the audio-only presentation, the audio-visual congruent presentation led to a more intense emotional response. More importantly, the audio-visual integration occurred both in the positive music and in the negative music. Furthermore, the audio-visual integration effect was larger for positive music than for negative music; meanwhile the audio-visual integration effect was strongest with the visual information presented within 80s for negative music, which indicated that this integration effect was more likely to occur in the negative music. These results suggest that when the music was positive, the effect of audio-visual integration was greater. When the music was negative, the modulation effect of the presentation durations of visual information on the music-induced emotion was more significant.

Increased Functional Brain Network Efficiency During Audiovisual Temporal Asynchrony Integration Task in Aging.

Audiovisual integration significantly changes over the lifespan, but age-related functional connectivity in audiovisual temporal asynchrony integration tasks remains underexplored. In the present study, electroencephalograms (EEGs) of 27 young adults (22–25 years) and 25 old adults (61–76 years) were recorded during an audiovisual temporal asynchrony integration task with seven conditions [auditory (A), visual (V), AV, A50V, A100V, V50A and V100A]. We calculated the phase lag index (PLI)-weighted connectivity networks modulated by the audiovisual tasks and found that the PLI connections showed obvious dynamic changes after stimulus onset. In the theta (4–7 Hz) and alpha (8–13 Hz) bands, the AV and V50A conditions induced stronger functional connections and higher global and local efficiencies, reflecting a stronger audiovisual integration effect, which was attributed to the auditory information arriving at the primary auditory cortex earlier than the visual information reaching the primary visual cortex. Importantly, the functional connectivity and network efficiencies of old adults revealed higher global and local efficiencies and higher degree in both the theta and alpha bands. These larger network efficiencies indicated that old adults might experience more difficulties in attention and cognitive control during the audiovisual integration task with temporal asynchrony than young adults. There were significant associations between network efficiencies and peak time of integration only in young adults. We propose that an audiovisual task with multiple conditions might arouse the appropriate attention in young adults but would lead to a ceiling effect in old adults. Our findings provide new insights into the network topography of old adults during audiovisual integration and highlight higher functional connectivity and network efficiencies due to greater cognitive demand.

Brain Responses to Letters and Speech Sounds and Their Correlations With Cognitive Skills Related to Reading in Children.

Letter-speech sound (LSS) integration is crucial for initial stages of reading acquisition. However, the relationship between cortical organization for supporting LSS integration, including unimodal and multimodal processes, and reading skills in early readers remains unclear. In the present study, we measured brain responses to Finnish letters and speech sounds from 29 typically developing Finnish children in a child-friendly audiovisual integration experiment using magnetoencephalography. Brain source activations in response to auditory, visual and audiovisual stimuli as well as audiovisual integration response were correlated with reading skills and cognitive skills predictive of reading development after controlling for the effect of age. Regression analysis showed that from the brain measures, the auditory late response around 400 ms showed the largest association with phonological processing and rapid automatized naming abilities. In addition, audiovisual integration effect was most pronounced in the left and right temporoparietal regions and the activities in several of these temporoparietal regions correlated with reading and writing skills. Our findings indicated the important role of temporoparietal regions in the early phase of learning to read and their unique contribution to reading skills.

Audiovisual integration in familiar person recognition

ABSTRACTInterpersonal communication is a multimodal experience. Perceptual integration of information conveyed by voice and face has previously been demonstrated for speech comprehension and speaker localization. For instance, audiovisual (McGurk) illusions occur when perceiving an auditory syllable with a time-synchronized incongruent syllable in a visual facial articulation. However, despite a large body of research on audiovisual integration (AVI), researchers have only recently begun to consider the role of AVI for person recognition. In the present paper, Experiment 1 demonstrated systematic benefits and costs for recognition of familiar voices when combined with time-synchronized articulating faces of corresponding or noncorresponding speaker identity, respectively. By contrast, no costs (relative to auditory-only presentation) occurred when voices were combined with static noncorresponding faces. Moreover, this pattern was clear for familiar but not unfamiliar voices. In Experiment 2, smaller benefits and costs for familiar voices when combined with time-reversed (as compared to time-synchronized) facial articulations suggest a role of synchrony for AVI, over and above effects of facial motion per se. Experiment 3 demonstrated benefits, as well as a trend for costs, of familiar face recognition when faces were presented with voices of corresponding or noncorresponding identity. Overall, these results suggest that AVI is an important factor in the recognition of people, depends on familiarity with a speaker, shows sensitivity to temporal synchronization of the facial and vocal articulation, and can occur in a bidirectional manner.

Visual field asymmetry in the facilitation effects of audio-visual integration on visual representation(Summary of Awarded Presentation at the 31st Annual Meeting)

Visual field asymmetry in the facilitation effects of audio-visual integration on visual representation(Summary of Awarded Presentation at the 31st Annual Meeting)

Audio-visual integration effect in lateral occipital cortex during an object recognition task: An interference pilot study.

Recent neuroimaging data support the hypothesis of a multisensory interplay at low-level sensory-specific cortex. We used an on-line interference approach by rTMS to investigate the role of the left lateral occipital cortex (LOC) in audio-visual (AV) object recognition process. Fifteen healthy volunteers performed a visual identification task of degraded pictures presented alone or simultaneously to coherent or non-coherent sounds. Focal 10-Hz rTMS at an intensity of 100% resting motor threshold was delivered simultaneously to the picture. Two blocks of 60 pictures were randomly displayed in two different experimental conditions: rTMS of the left LOC and over Cz. rTMS of the left LOC produced a worsening of the accuracy compared to rTMS over Cz specifically in the coherent AV condition. These data support the view that audio-visual interaction effect may occur at early stage of recognition processing.

Benefits for Voice Learning Caused by Concurrent Faces Develop over Time

Recognition of personally familiar voices benefits from the concurrent presentation of the corresponding speakers’ faces. This effect of audiovisual integration is most pronounced for voices combined with dynamic articulating faces. However, it is unclear if learning unfamiliar voices also benefits from audiovisual face-voice integration or, alternatively, is hampered by attentional capture of faces, i.e., “face-overshadowing”. In six study-test cycles we compared the recognition of newly-learned voices following unimodal voice learning vs. bimodal face-voice learning with either static (Exp. 1) or dynamic articulating faces (Exp. 2). Voice recognition accuracies significantly increased for bimodal learning across study-test cycles while remaining stable for unimodal learning, as reflected in numerical costs of bimodal relative to unimodal voice learning in the first two study-test cycles and benefits in the last two cycles. This was independent of whether faces were static images (Exp. 1) or dynamic videos (Exp. 2). In both experiments, slower reaction times to voices previously studied with faces compared to voices only may result from visual search for faces during memory retrieval. A general decrease of reaction times across study-test cycles suggests facilitated recognition with more speaker repetitions. Overall, our data suggest two simultaneous and opposing mechanisms during bimodal face-voice learning: while attentional capture of faces may initially impede voice learning, audiovisual integration may facilitate it thereafter.

The Effect of Audiovisual Integration on Performance Accuracy and Learning in Motor Task

Introduction: Vision is not the only sense providing information to understand the pattern of movement. Hearing is another suitable channel to gather information about the pattern of movement. Hence, this study investigated the effect of audiovisual integration on performance precision and learning in basketball jump shot. Materials and Methods: 30 subjects were selected. The subjects were randomly divided three groups: visual, visual-auditory (single channel) and visual-auditory (two channels). Visual group watched patterns killed basketball player and other groups other groups in addition to watching pattern skilled basketball player, heard Elbow angular velocity (one channel groups) and Elbow and wrist angular velocity (two channel groups) as sonification. Per session practice was including 40 try; and subject participated in four sessions . The acquisition test was done at the end of each session and retention test was performed after 48 hours. Data was analyzed by SPSS version 22. One-way ANOVA and repeated measures ANOVA was used for data analysis. Results: Results showed Visual-auditory group (single channel) had significantly better performance compared to visual group (P = 0.001) and visual-auditory group (two channels) (P = 0.045) inacquisition test. In retention test there was significant difference between auditory groups and visual group in favor of the auditory groups (P = 0.002 and P = 0.001, respectively). Conclusion: The results are consistent with modality appropriateness hypothesis. This results argues that due to certain temporal adjustments in jump shot basketball task, modality that dominates the perception in the context of this task, is auditory when is presented with visual simultaneously, it provide beneficial results. In this study, positive effect visual-auditory integration on performance and learning was confirmed.

The integration of facial and vocal cues during emotional change perception: EEG markers

The ability to detect emotional changes is of primary importance for social living. Though emotional signals are often conveyed by multiple modalities, how emotional changes in vocal and facial modalities integrate into a unified percept has yet to be directly investigated. To address this issue, we asked participants to detect emotional changes delivered by facial, vocal and facial-vocal expressions while behavioral responses and electroencephalogram were recorded. Behavioral results showed that bimodal emotional changes were detected with higher accuracy and shorter response latencies compared with each unimodal condition. Moreover, the detection of emotional change, regardless of modalities, was associated with enhanced amplitudes in the N2 and P3 component, as well as greater theta synchronization. More importantly, the P3 amplitudes and theta synchronization were larger for the bimodal emotional change condition than for the sum of the two unimodal conditions. The superadditive responses in P3 amplitudes and theta synchronization were both positively correlated with the magnitude of the bimodal superadditivity in accuracy. These behavioral and electrophysiological data consistently illustrated an effect of audiovisual integration during the detection of emotional changes, which is most likely mediated by the P3 activity and theta oscillations in brain responses.