Audiovisual Interactions Research Articles

Age-related differences in audiovisual interactions of semantically different stimuli.

Converging results have shown that adults benefit from congruent multisensory stimulation in the identification of complex stimuli, whereas the developmental trajectory of the ability to integrate multisensory inputs in children is less well understood. In this study we explored the effects of audiovisual semantic congruency on identification of visually presented stimuli belonging to different categories, using a cross-modal approach. Four groups of children ranging in age from 6 to 13 years and adults were administered an object identification task of visually presented pictures belonging to living and nonliving entities. Stimuli were presented in visual, congruent audiovisual, incongruent audiovisual, and noise conditions. Results showed that children under 12 years of age did not benefit from multisensory presentation in speeding up the identification. In children the incoherent audiovisual condition had an interfering effect, especially for the identification of living things. These data suggest that the facilitating effect of the audiovisual interaction into semantic factors undergoes developmental changes and the consolidation of adult-like processing of multisensory stimuli begins in late childhood. (PsycINFO Database Record

Sounds Are Perceived as Louder When Accompanied by Visual Movement

In this study, we present three experiments investigating the influence of visual movement on auditory judgements. In Experiments 1 and 2, two bursts of noise were presented and participants were required to judge which was louder using a forced-choice task. One of the two bursts was accompanied by a moving disc. The other burst either was accompanied by no visual stimulus (Experiment 1) or by a static disc (Experiment 2). When the two sounds were of identical intensity participants judged the sound accompanied by the moving disc as louder. The effect was greater when auditory stimuli were of the same intensity but it was still present for mid-to-high intensities. In a third, control, experiment participants judged the pitch (and not the loudness) of a pair of tones. Here the pattern was different: there was no effect of visual motion for sounds of the same pitch, with a reversed effect for mid-to-high pitch differences (the effect of motion lowered the pitch). This showed no shift of response towards the interval accompanied by the moving disc. In contrast, the effect on pitch was reversed in comparison to what observed for loudness, with mid-to-high frequency sound accompanied by motion rated as lower in pitch respect to the static intervals.The natural tendency for moving objects to elicit sounds may lead to an automatic perceptual influence of vision over sound particularly when the latter is ambiguous. This is the first account of this novel audio-visual interaction.

La comprobación de los canales al inicio de encuentros por videoconferencia

Partiendo de la metodología del Análisis Conversacional, el presente estudio examina una práctica habitual durante el inicio de encuentros por videoconferencia: la comprobación de los canales de audio y vídeo. Se muestra que dicha práctica con frecuencia se lleva a cabo a través de elementos comoholaacompañados de marcas prosódicas concretas en función de las cuales los participantes pueden realizar diferentes acciones: solicitar una confirmación, confirmar el correcto funcionamiento de los canales o marcar la posible existencia de problemas técnicos. Se argumenta que aunque con dichos elementos también se contribuye al establecimiento de la orientación mutua y la disponibilidad recíproca de los participantes, la función principal de la primera secuencia de la interacción audiovisual es el chequeo del funcionamiento de los canales, por lo que se concluye que en videoconferencia la secuencia inicial de comprobación podría constituir la secuencia típica con la que se inicia la interacción oral.

Migrant Parenting and Mobile Phone Use: Building Quality Relationships between Chinese Migrant Workers and their Left-behind Children

This study examines the uses and derived gratifications of mobile phones among migrant parents in communicating with children they left behind. Data were gathered from a sample of 378 migrant parents who worked in factories in southern China. The results show that migrant workers who used mobile phones to assist in distance parenting were motivated by a desire for instantaneous communication (e.g., immediate access and reassurance), online transactions, affection, mobility, relaxation, and information. The demographic results show that the migrant workers tended to communicate with their older male children via calls and text messaging with their female children via audiovisual interaction. When migrant parents called, texted, or interacted with their children via audiovisual functions, they were motivated by the need for affection and relaxation. In predicting the quality of migrant workers’ involvement in parenting, the current study found that significant motives for the use of mobile phones included their ability to care for the children they left behind through communicating instantaneously, expressing affection, and conducting online transactions. The current paper discusses the theoretical and practical implications of the results of this study.

Temporal and spectral audiotactile interactions in musicians.

Previous investigations have revealed that the complex sensory exposure of musical training alters audiovisual interactions. As of yet, there has been little evidence on the effects of musical training on audiotactile interactions at a behavioural level. Here, we tested audiotactile interaction in musicians using the audiotactile illusory flash and the parchment-skin illusion. Significant differences were only found between musicians and non-musicians for the audiotactile illusory flash. Both groups had similar task-relevant unisensory abilities, but unlike non-musicians, the number of auditory stimulations did not have a statistically important influence on the number of perceived tactile stimulations for musicians. Musicians and non-musicians similarly perceived the parchment-skin illusion. Spectral alterations of self-generated palmar sounds similarly altered the perception of wetness and dryness for both groups. These results suggest that musical training does not seem to alter multisensory interactions at large. The specificity of the sensory enhancement suggests that musical training specifically alters processes underlying the interaction of temporal audiotactile stimuli and not the global interaction between these modalities. These results are consistent with previous unisensory and multisensory investigations on sensory abilities related to audiotactile processing in musicians.

Playing visual dominance of score on the piano: Skilled motor action matters in the awareness of musical notes during binocular rivalry, only when accompanied by auditory feedback

Previous studies in our group showed that a visual musical score viewed dichoptically with a radial grating enjoyed longer dominance durations when presented with an auditory melody congruent to that score (Kim et al., VSS 2014; Lee et al., 2015). It has also been reported that an action relevant to a visual interpretation contributes to resolve perceptual ambiguity during binocular rivalry (Beets et al., 2010; Maruya et al., 2007). In the present study, we investigated whether a skilled motor action linked tightly to one of the two rival targets affects audiovisual interaction during rivalry. Sixteen observers with varying degree of piano playing skill viewed dichoptically a musical score scrolling from right to left within a viewing window and a counter-phase flickering radial grating. The perceptual dominance of visual musical score was tracked by playing the musical notes on the midi keyboard, while the grating dominance was tracked by pressing a button on a computer keyboard. On "sound" trials, participants heard the sound of piano being played by themselves. On "no sound" trials, auditory feedback was not accompanied by keyboard playing. Results showed a positive correlation between the degree of piano playing skill and the normalized score-dominance durations, only when the auditory feedback was accompanied. Accordingly, observers were divided in two groups based on the skill. In the more-trained group (8.3 ±0.89 years in training, N=8), the normalized score-dominance durations with auditory feedback were distributed toward the longer side than without auditory feedback, while there was no such effect observed in the less-trained group (3.6 ± 0.56 years, N=8). These results suggest that an execution of skilled motor action closely linked to a visual stimulus has impact on the interaction between the visual stimuli and the auditory feedback during binocular rivalry, which relies on the mastery of the motor skill. Meeting abstract presented at VSS 2016

Visual Mislocalization of Moving Objects in an Audiovisual Event.

The present study investigated the influence of an auditory tone on the localization of visual objects in the stream/bounce display (SBD). In this display, two identical visual objects move toward each other, overlap, and then return to their original positions. These objects can be perceived as either streaming through or bouncing off each other. In this study, the closest distance between object centers on opposing trajectories and tone presentation timing (none, 0 ms, ± 90 ms, and ± 390 ms relative to the instant for the closest distance) were manipulated. Observers were asked to judge whether the two objects overlapped with each other and whether the objects appeared to stream through, bounce off each other, or reverse their direction of motion. A tone presented at or around the instant of the objects’ closest distance biased judgments toward “non-overlapping,” and observers overestimated the physical distance between objects. A similar bias toward direction change judgments (bounce and reverse, not stream judgments) was also observed, which was always stronger than the non-overlapping bias. Thus, these two types of judgments were not always identical. Moreover, another experiment showed that it was unlikely that this observed mislocalization could be explained by other previously known mislocalization phenomena (i.e., representational momentum, the Fröhlich effect, and a turn-point shift). These findings indicate a new example of crossmodal mislocalization, which can be obtained without temporal offsets between audiovisual stimuli. The mislocalization effect is also specific to a more complex stimulus configuration of objects on opposing trajectories, with a tone that is presented simultaneously. The present study promotes an understanding of relatively complex audiovisual interactions beyond simple one-to-one audiovisual stimuli used in previous studies.

Audio-visual integration effect in lateral occipital cortex during an object recognition task: An interference pilot study.

Recent neuroimaging data support the hypothesis of a multisensory interplay at low-level sensory-specific cortex. We used an on-line interference approach by rTMS to investigate the role of the left lateral occipital cortex (LOC) in audio-visual (AV) object recognition process. Fifteen healthy volunteers performed a visual identification task of degraded pictures presented alone or simultaneously to coherent or non-coherent sounds. Focal 10-Hz rTMS at an intensity of 100% resting motor threshold was delivered simultaneously to the picture. Two blocks of 60 pictures were randomly displayed in two different experimental conditions: rTMS of the left LOC and over Cz. rTMS of the left LOC produced a worsening of the accuracy compared to rTMS over Cz specifically in the coherent AV condition. These data support the view that audio-visual interaction effect may occur at early stage of recognition processing.

When audiovisual correspondence disturbs visual processing

Multisensory integration is known to create a more robust and reliable perceptual representation of one's environment. Specifically, a congruent auditory input can make a visual stimulus more salient, consequently enhancing the visibility and detection of the visual target. However, it remains largely unknown whether a congruent auditory input can also impair visual processing. In the current study, we demonstrate that temporally congruent auditory input disrupts visual processing, consequently slowing down visual target detection. More importantly, this cross-modal inhibition occurs only when the contrast of visual targets is high. When the contrast of visual targets is low, enhancement of visual target detection is observed, consistent with the prediction based on the principle of inverse effectiveness (PIE) in cross-modal integration. The switch of the behavioral effect of audiovisual interaction from benefit to cost further extends the PIE to encompass the suppressive cross-modal interaction.

Age-related audiovisual interactions in the superior colliculus of the rat

It is well established that multisensory integration is a functional characteristic of the superior colliculus that disambiguates external stimuli and therefore reduces the reaction times toward simple audiovisual targets in space. However, in a condition where a complex audiovisual stimulus is used, such as the optical flow in the presence of modulated audio signals, little is known about the processing of the multisensory integration in the superior colliculus. Furthermore, since visual and auditory deficits constitute hallmark signs during aging, we sought to gain some insight on whether audiovisual processes in the superior colliculus are altered with age. Extracellular single-unit recordings were conducted in the superior colliculus of anesthetized Sprague–Dawley adult (10–12months) and aged (21–22months) rats. Looming circular concentric sinusoidal (CCS) gratings were presented alone and in the presence of sinusoidally amplitude modulated white noise. In both groups of rats, two different audiovisual response interactions were encountered in the spatial domain: superadditive, and suppressive. In contrast, additive audiovisual interactions were found only in adult rats. Hence, superior colliculus audiovisual interactions were more numerous in adult rats (38%) than in aged rats (8%). These results suggest that intersensory interactions in the superior colliculus play an essential role in space processing toward audiovisual moving objects during self-motion. Moreover, aging has a deleterious effect on complex audiovisual interactions.

Audio-visual interaction of environmental noise

Currently research into the psychological evaluation of noise in daily life is carried out without taking into account the sense of sight. The human senses interact with each other; thus some information coming from one sense can be skipped or ignored in favor of information coming from another sense, leading to completely different reactions or behavior. The aim of this paper is to verify, on the basis of psychophysical experiments, how a human being processes audio-visual information coming from the different environmental noises which can be encountered in daily life. The experiment was divided into three parts: auditory, visual, and audio-visual. In each part of the experiment, the ICBEN scale (0-10) was used to rate the presented stimuli. In the first part only audio stimuli were presented, and subjects were asked to rate their annoyance with the sound. In the second part of the experiment, the participants were asked to rate how pleasant the presented video clips were. Finally, in the last part of the experiment, participants were presented with a compatible and incompatible mix of audio and visual stimuli and asked to rate their annoyance. We found that several audio stimuli were assessed differently, to a significant extent, by listeners after video clips were added to them.

Can you hear me yet? An intracranial investigation of speech and non-speech audiovisual interactions in human cortex

ABSTRACTIn everyday conversation, viewing a talker's face can provide information about the timing and content of an upcoming speech signal, resulting in improved intelligibility. Using electrocorticography, we tested whether human auditory cortex in Heschl's gyrus (HG) and on superior temporal gyrus (STG) and motor cortex on precentral gyrus (PreC) were responsive to visual/gestural information prior to the onset of sound and whether early stages of auditory processing were sensitive to the visual content (speech syllable versus non-speech motion). Event-related band power (ERBP) in the high gamma band was content-specific prior to acoustic onset on STG and PreC, and ERBP in the beta band differed in all three areas. Following sound onset, we found no evidence for content-specificity in HG, evidence for visual specificity in PreC, and specificity for both modalities in STG. These results support models of audiovisual processing in which sensory information is integrated in non-primary cortical areas.

Anatomic and Functional Connectivity Relationship in Autistic Children During Three Different Experimental Conditions.

A group of 21 autistic children were studied for determining the relationship between the anatomic (AC) versus functional (FC) connectivity, considering short-range and long-range brain networks. AC was assessed by the DW-MRI technique and FC by EEG coherence calculation, in three experimental conditions: basal, watching a popular cartoon with audio (V-A), and with muted audio track (VwA). For short-range connections, basal records, statistical significant correlations were found for all EEG bands in the left hemisphere, but no significant correlations were found for fast EEG frequencies in the right hemisphere. For the V-A condition, significant correlations were mainly diminished for the left hemisphere; for the right hemisphere, no significant correlations were found for the fast EEG frequency bands. For the VwA condition, significant correlations for the rapid EEG frequencies mainly disappeared for the right hemisphere. For long-range connections, basal records showed similar correlations for both hemispheres. For the right hemisphere, significant correlations incremented to all EEG bands for the V-A condition, but these significant correlations disappeared for the fast EEG frequencies in the VwA condition. It appears that in a resting-state condition, AC is better associated with functional connectivity for short-range connections in the left hemisphere. The V-A experimental condition enriches the AC and FC association for long-range connections in the right hemisphere. This might be related to an effective connectivity improvement due to full video stimulation (visual and auditory). An impaired audiovisual interaction in the right hemisphere might explain why significant correlations disappeared for the fast EEG frequencies in the VwA experimental condition.

Enhanced audio–visual interactions in the auditory cortex of elderly cochlear-implant users

Auditory deprivation and the restoration of hearing via a cochlear implant (CI) can induce functional plasticity in auditory cortical areas. How these plastic changes affect the ability to integrate combined auditory (A) and visual (V) information is not yet well understood. In the present study, we used electroencephalography (EEG) to examine whether age, temporary deafness and altered sensory experience with a CI can affect audio–visual (AV) interactions in post-lingually deafened CI users. Young and elderly CI users and age-matched NH listeners performed a speeded response task on basic auditory, visual and audio–visual stimuli. Regarding the behavioral results, a redundant signals effect, that is, faster response times to cross-modal (AV) than to both of the two modality-specific stimuli (A, V), was revealed for all groups of participants. Moreover, in all four groups, we found evidence for audio–visual integration. Regarding event-related responses (ERPs), we observed a more pronounced visual modulation of the cortical auditory response at N1 latency (approximately 100 ms after stimulus onset) in the elderly CI users when compared with young CI users and elderly NH listeners. Thus, elderly CI users showed enhanced audio–visual binding which may be a consequence of compensatory strategies developed due to temporary deafness and/or degraded sensory input after implantation. These results indicate that the combination of aging, sensory deprivation and CI facilitates the coupling between the auditory and the visual modality. We suggest that this enhancement in multisensory interactions could be used to optimize auditory rehabilitation, especially in elderly CI users, by the application of strong audio-visually based rehabilitation strategies after implant switch-on.

Policing Fish at Boston's Museum of Science: Studying Audiovisual Interaction in the Wild.

Boston's Museum of Science supports researchers whose projects advance science and provide educational opportunities to the Museum's visitors. For our project, 60 visitors to the Museum played “Fish Police!!,” a video game that examines audiovisual integration, including the ability to ignore irrelevant sensory information. Players, who ranged in age from 6 to 82 years, made speeded responses to computer-generated fish that swam rapidly across a tablet display. Responses were to be based solely on the rate (6 or 8 Hz) at which a fish's size modulated, sinusoidally growing and shrinking. Accompanying each fish was a task-irrelevant broadband sound, amplitude modulated at either 6 or 8 Hz. The rates of visual and auditory modulation were either Congruent (both 6 Hz or 8 Hz) or Incongruent (6 and 8 or 8 and 6 Hz). Despite being instructed to ignore the sound, players of all ages responded more accurately and faster when a fish's auditory and visual signatures were Congruent. In a controlled laboratory setting, a related task produced comparable results, demonstrating the robustness of the audiovisual interaction reported here. Some suggestions are made for conducting research in public settings.

MHealth Clinic Appointment PC Tablet: Implementation, Challenges and Solutions.

Patients requiring daily intravenous (IV) home parenteral nutrition (HPN) would benefit from in-home professional observation to improve self-care, to assess, detect and prevent serious complications. The study aims are to assess the viability and utility of conducting mobile healthcare (mHealth) videoconference assessments with patients managing lifelong daily 12-hour IV nutrition infusions in their homes. The challenges and solutions to implementing mobile personal computer (PC) tablet based clinic appointments are described. A wireless Apple iPad Mini™ mobile touch-screen tablet computer with 5 mega-pixel camera was loaned to patients. Each tablet had Polycom RealPresence software and a fourth generation (4G) mobile telecommunications data plan. These supported audio-visual mobile videoconferencing encrypted connections between health professionals in their offices and HPN patients and their family members in their homes. Patients' and professionals' evaluations of their mHealth clinic experiences are collected. Patients (mean age = 41.9, SD = 2.8 years) had been prescribed 12-hour home parenteral nutrition (HPN) infusions daily due short bowel disorders. Patients had been on HPN from 1 to 10 years (M=4, SD=3.6). Evaluation of clinic appointments revealed that 100% of the patients (n=45) and the professionals (n=6) indicated that they can clearly hear and easily see one another. The mHealth audio-visual interactions were highly rated by patients and family members. Professionals highly rated their ability to obtain a medical history and visual inspection of patients. Several challenges were identified and recommendations for resolutions are described. All patients and professionals highly rated the iPad mHealth clinic appointments for convenience and ease of communicating between homes and offices. An important challenge for all mHealth visits is the clinical professional's ability to make clinically accurate judgments about what they observed and heard from the patients. Following our solutions for obtaining clear visuals with the iPad can improve ability to make clinical assessments.

Audio–visual integration through the parallel visual pathways

Audio–visual integration has been shown to be present in a wide range of different conditions, some of which are processed through the dorsal, and others through the ventral visual pathway. Whereas neuroimaging studies have revealed integration-related activity in the brain, there has been no imaging study of the possible role of segregated visual streams in audio–visual integration.We set out to determine how the different visual pathways participate in this communication. We investigated how audio–visual integration can be supported through the dorsal and ventral visual pathways during the double flash illusion. Low-contrast and chromatic isoluminant stimuli were used to drive preferably the dorsal and ventral pathways, respectively. In order to identify the anatomical substrates of the audio–visual interaction in the two conditions, the psychophysical results were correlated with the white matter integrity as measured by diffusion tensor imaging.The psychophysiological data revealed a robust double flash illusion in both conditions. A correlation between the psychophysical results and local fractional anisotropy was found in the occipito-parietal white matter in the low-contrast condition, while a similar correlation was found in the infero-temporal white matter in the chromatic isoluminant condition. Our results indicate that both of the parallel visual pathways may play a role in the audio–visual interaction.

Melodic sound enhances visual awareness of congruent musical notes, but only if you can read music

Predictive influences of auditory information on resolution of visual competition were investigated using music, whose visual symbolic notation is familiar only to those with musical training. Results from two experiments using different experimental paradigms revealed that melodic congruence between what is seen and what is heard impacts perceptual dynamics during binocular rivalry. This bisensory interaction was observed only when the musical score was perceptually dominant, not when it was suppressed from awareness, and it was observed only in people who could read music. Results from two ancillary experiments showed that this effect of congruence cannot be explained by differential patterns of eye movements or by differential response sluggishness associated with congruent score/melody combinations. Taken together, these results demonstrate robust audiovisual interaction based on high-level, symbolic representations and its predictive influence on perceptual dynamics during binocular rivalry.

Sound can suppress visual perception.

In a single modality, the percept of an input (e.g., voices of neighbors) is often suppressed by another (e.g., the sound of a car horn nearby) due to close interactions of neural responses to these inputs. Recent studies have also suggested that close interactions of neural responses could occur even across sensory modalities, especially for audio-visual interactions. However, direct behavioral evidence regarding the audio-visual perceptual suppression effect has not been reported in a study with humans. Here, we investigated whether sound could have a suppressive effect on visual perception. We found that white noise bursts presented through headphones degraded visual orientation discrimination performance. This auditory suppression effect on visual perception frequently occurred when these inputs were presented in a spatially and temporally consistent manner. These results indicate that the perceptual suppression effect could occur across auditory and visual modalities based on close and direct neural interactions among those sensory inputs.

Audio-visual interactions in environment assessment

The aim of the study was to examine how visual and audio information influences audio-visual environment assessment. Original audio-visual recordings were made at seven different places in the city of Poznań. Participants of the psychophysical experiments were asked to rate, on a numerical standardized scale, the degree of comfort they would feel if they were in such an environment. The assessments of audio-visual comfort were carried out in a laboratory in four different conditions: (a) audio samples only, (b) original audio-visual samples, (c) video samples only, and (d) mixed audio-visual samples. The general results of this experiment showed a significant difference between the investigated conditions, but not for all the investigated samples. There was a significant improvement in comfort assessment when visual information was added (in only three out of 7 cases), when conditions (a) and (b) were compared. On the other hand, the results show that the comfort assessment of audio-visual samples could be changed by manipulating the audio rather than the video part of the audio-visual sample. Finally, it seems, that people could differentiate audio-visual representations of a given place in the environment based rather of on the sound sources' compositions than on the sound level. Object identification is responsible for both landscape and soundscape grouping.