Audiovisual Processing Research Articles

Integrating Visual Information into the Auditory Cortex Promotes Sound Discrimination through Choice-Related Multisensory Integration.

An increasing number of studies have shown that cross-modal interaction can occur in early sensory cortices. Yet, how neurons in sensory cortices integrate multisensory cues in perceptual tasks and to what extent this influences behavior is largely unclear. To investigate, we examined visual modulation of auditory responses in the primary auditory cortex (A1) in a two-alternative forced-choice task. During the task, male rats were required to make a behavioral choice based on the pure tone frequency (low vs high) of the self-triggered stimulus to get a water reward. The result showed that the presence of a noninformative visual cue did not uniformly influence auditory response, with frequently enhancing just one of them. Closely correlated with behavioral choice, the visual cue mainly enhanced responsiveness to the auditory cue indicating a movement direction contralateral to A1 being recorded. Operating in this fashion provided A1 neurons a superior capability to discriminate sound during multisensory trials. Concomitantly, behavioral data and decoding analysis revealed that visual cue presence could speed the process of sound discrimination. We also observed this differential multisensory integration effect in well-trained rats when tested with passive stimulation and under anesthesia, albeit to a much lesser extent. We did not see this differentially integrative effect while recording in A1 in another similar group of rats performing a free-choice task. These data suggest that auditory cortex can engage in meaningful audiovisual processing, and perceptual learning can modify its multisensory integration mechanism to meet task requirements.SIGNIFICANCE STATEMENT In the natural environment, visual stimuli are frequently accompanied by auditory cues. Although multisensory integration has traditionally been seen as a feature of associational cortices, recent studies have shown that cross-modal inputs can also influence neuronal activity in primary sensory cortices. However, exactly how neurons in sensory cortices integrate multisensory cues to guide behavioral choice is still unclear. Here, we describe a novel model of multisensory integration used by A1 neurons to shape auditory representations when rats performed a cue-guided task. We found that a task-irrelevant visual cue could specifically enhance the response of neurons in sound guiding to the contralateral choice. This differentially integrative model facilitated sound discrimination and behavioral choice. This result indicates that task engagement can modulate multisensory integration.

Introducing Cognitive Psychology in Film Studies: Redefining Affordance

This paper investigates how interdisciplinary research impacts the film industry in research and practice by introducing psychological concepts. Psychology, especially neural and cognitive science, provides a distinct advantage when examining humans’ audio-visual processing mechanisms and esthetics questions regarding the film. By introducing psychology, film researchers and filmmakers could rethink and evaluate the current research paradigm from a broader point of view. This paper consists of three parts: (1) a discussion on the nature of film using an interdisciplinary approach; (2) a discussion on the characteristics and attributes of film; (3) an introduction of the psychological concept of “affordance” to film studies and practice. Although the film interdisciplinary research paradigm is still under development, we argue that introducing the other subjects is innovating the field of film research, providing us with a new angle to examine the intersections of ubiquitous but complex human esthetics activities.

Bioshock: Infinite as the Mirror of America?

Few mainstream computer games have caused such controversy as Bioshock: Infinite (2013). The third instalment of the Bioshock series is set in the fictional city of Columbia in an alternate history of the early twentieth century, which at first glance appears to be a perfect social utopia. After a while, however, the narrative begins to uncover the multilayered problems of society oppressed by a fraction of the white elite and religious fanaticism. The popularity of the game is not only due to the attractive audiovisual processing and complex game mechanics. It is primarily a story that uses an unprecedented amount of religious symbolism – especially Christian symbols, historical references, polysemic story elements, and a story based on the concepts of frontier myth and American exceptionalism. This work will deal with analysing these phenomena, especially those directly related to the religious and nationalistic topics in the United States.

Code-switched end-to-end Marathi speech recognition for especially abled people

There is no need for eyes or arms in speech communication. Instead, both humans and machines communicate by speaking and listening. Therefore, speech communication not only helps physically disabled but also help people with hearing loss and/or language disorders. It is also used by people with vision or motor disabilities. There are various types of devices that are used for people with hearing loss. They can be classified as audio-visual stimulation devices, tactile devices, and speech processing devices. These devices consider the characteristics of speech and/or deploy automatic speech recognition inside it. This article focuses on such ASR system based on E2E model which can be useful in such devices. This research is carried out on the language used by the families of India’s over 80 million speakers i.e., Marathi. These assessments provide valuable insights into the development of SR models for low-resourced Indian languages. We present E2E ASR model trained with code-switched text of Marathi language and decoded similarly to a normal E2E ASR system. This model achieves a better word error rate of 35.09%, according to experimental results on the code-switched Marathi corpus. We compared our model to Facebook’s wav2vec2 model, which produces a WER of 23.72 percent on code-switched Marathi datasets without any language model and 12.10 percent on open ASR Marathi datasets.

Pandemics, Delays, and Pure Data: on ‘afterlives’ (2020), for Flute and Live Electronics and Visuals

The essay addresses creative and technical aspects of the piece ‘afterlives’ (2020), for flute and live electronics and visuals. Composed and premiered in the context of the COVID-19 pandemic, the composition employs audiovisual processes based on different audiovisual techniques: phase-vocoders, buffer-based granulations, Ambisonics spatialization, and variable delay of video streams. The resulting sounds and images allude to typical situations of social interaction via video conferencing applications. ‘Afterlives’ relies on an interplay between current, almost-current, and past moments of the audiovisual streams, which dephase the performer’s images and sounds. I have avoided, in text, delving deeper into the Pure Data abstractions and or into the musical analysis of my composition. The main purpose of the text is rather to present compositional/technical elements of ‘afterlives’ and discuss how they enable new experiences of time.

Developmental Trajectories of Letter and Speech Sound Integration During Reading Acquisition.

Reading acquisition in alphabetic languages starts with learning the associations between speech sounds and letters. This learning process is related to crucial developmental changes of brain regions that serve visual, auditory, multisensory integration, and higher cognitive processes. Here, we studied the development of audiovisual processing and integration of letter-speech sound pairs with an audiovisual target detection functional MRI paradigm. Using a longitudinal approach, we tested children with varying reading outcomes before the start of reading acquisition (T1, 6.5 yo), in first grade (T2, 7.5 yo), and in second grade (T3, 8.5 yo). Early audiovisual integration effects were characterized by higher activation for incongruent than congruent letter-speech sound pairs in the inferior frontal gyrus and ventral occipitotemporal cortex. Audiovisual processing in the left superior temporal gyrus significantly increased from the prereading (T1) to early reading stages (T2, T3). Region of interest analyses revealed that activation in left superior temporal gyrus (STG), inferior frontal gyrus and ventral occipitotemporal cortex increased in children with typical reading fluency skills, while poor readers did not show the same development in these regions. The incongruency effect bilaterally in parts of the STG and insular cortex at T1 was significantly associated with reading fluency skills at T3. These findings provide new insights into the development of the brain circuitry involved in audiovisual processing of letters, the building blocks of words, and reveal early markers of audiovisual integration that may be predictive of reading outcomes.

ERP Correlates of Altered Orthographic-Phonological Processing in Dyslexia.

Automatic visual word recognition requires not only well-established phonological and orthographic representations but also efficient audio-visual integration of these representations. One possibility is that in developmental dyslexia, inefficient orthographic processing might underlie poor reading. Alternatively, reading deficit could be due to inefficient phonological processing or inefficient integration of orthographic and phonological information. In this event-related potential study, participants with dyslexia (N = 25) and control readers (N = 27) were presented with pairs of words and pseudowords in an implicit same-different task. The reference-target pairs could be identical, or different in the identity or the position of the letters. To test the orthographic-phonological processing, target stimuli were presented in visual-only and audiovisual conditions. Participants with and without dyslexia processed the reference stimuli similarly; however, group differences emerged in the processing of target stimuli, especially in the audiovisual condition where control readers showed greater N1 responses for words than for pseudowords, but readers with dyslexia did not show such difference. Moreover, after 300 ms lexicality effect exhibited a more focused frontal topographic distribution in readers with dyslexia. Our results suggest that in developmental dyslexia, phonological processing and audiovisual processing deficits are more pronounced than orthographic processing deficits.

Topographical and laminar distribution of audiovisual processing within human planum temporale.

The brain is capable of integrating signals from multiple sensory modalities. Such multisensory integration can occur in areas that are commonly considered unisensory, such as planum temporale (PT) representing the auditory association cortex. However, the roles of different afferents (feedforward vs. feedback) to PT in multisensory processing are not well understood. Our study aims to understand that by examining laminar activity patterns in different topographical subfields of human PT under unimodal and multisensory stimuli. To this end, we adopted an advanced mesoscopic (sub-millimeter) fMRI methodology at 7 T by acquiring BOLD (blood-oxygen-level-dependent contrast, which has higher sensitivity) and VAPER (integrated blood volume and perfusion contrast, which has superior laminar specificity) signal concurrently, and performed all analyses in native fMRI space benefiting from an identical acquisition between functional and anatomical images. We found a division of function between visual and auditory processing in PT and distinct feedback mechanisms in different subareas. Specifically, anterior PT was activated more by auditory inputs and received feedback modulation in superficial layers. This feedback depended on task performance and likely arose from top-down influences from higher-order multimodal areas. In contrast, posterior PT was preferentially activated by visual inputs and received visual feedback in both superficial and deep layers, which is likely projected directly from the early visual cortex. Together, these findings provide novel insights into the mechanism of multisensory interaction in human PT at the mesoscopic spatial scale.

Audiovisual Interactions Among Near-Threshold Oscillating Stimuli in the Far Periphery Are Phase-Dependent.

Recent studies have highlighted the possible contributions of direct connectivity between early sensory cortices to audiovisual integration. Anatomical connections between the early auditory and visual cortices are concentrated in visual sites representing the peripheral field of view. Here, we aimed to engage early sensory interactive pathways with simple, far-peripheral audiovisual stimuli (auditory noise and visual gratings). Using a modulation detection task in one modality performed at an 84% correct threshold level, we investigated multisensory interactions by simultaneously presenting weak stimuli from the other modality in which the temporal modulation was barely-detectable (at 55 and 65% correct detection performance). Furthermore, we manipulated the temporal congruence between the cross-sensory streams. We found evidence for an influence of barely-detectable visual stimuli on the response times for auditory stimuli, but not for the reverse effect. These visual-to-auditory influences only occurred for specific phase-differences (at onset) between the modulated audiovisual stimuli. We discuss our findings in the light of a possible role of direct interactions between early visual and auditory areas, along with contributions from the higher-order association cortex. In sum, our results extend the behavioral evidence of audio-visual processing to the far periphery, and suggest – within this specific experimental setting – an asymmetry between the auditory influence on visual processing and the visual influence on auditory processing.

Incongruence effects in cross-modal emotional processing in autistic traits: An fMRI study

In everyday life, emotional information is often conveyed by both the face and the voice. Consequently, information presented by one source can alter the way in which information from the other source is perceived, leading to emotional incongruence. Here, we used functional magnetic resonance imaging (fMRI) to examine neutral correlates of two different types of emotional incongruence in audiovisual processing, namely incongruence of emotion-valence and incongruence of emotion-presence. Participants were in two groups, one group with a low Autism Quotient score (LAQ) and one with a high score (HAQ). Each participant experienced emotional (happy, fearful) or neutral faces or voices while concurrently being exposed to emotional (happy, fearful) or neutral voices or faces. They were instructed to attend to either the visual or auditory track. The incongruence effect of emotion-valence was characterized by activation in a wide range of brain regions in both hemispheres involving the inferior frontal gyrus, cuneus, superior temporal gyrus, and middle frontal gyrus. The incongruence effect of emotion-presence was characterized by activation in a set of temporal and occipital regions in both hemispheres, including the middle occipital gyrus, middle temporal gyrus and inferior temporal gyrus. In addition, the present study identified greater recruitment of the right inferior parietal lobule in perceiving audio-visual emotional expressions in HAQ individuals, as compared to the LAQ individuals. Depending on face or voice-to-be attended, different patterns of emotional incongruence were found between the two groups. Specifically, the HAQ group tend to show more incidental processing to visual information whilst the LAQ group tend to show more incidental processing to auditory information during the crossmodal emotional incongruence decoding. These differences might be attributed to different attentional demands and different processing strategies between the two groups.

Somatosensory contribution to audio-visual speech processing

Recent studies have demonstrated that the auditory speech perception of a listener can be modulated by somatosensory input applied to the facial skin suggesting that perception is an embodied process. However, speech perception is a multisensory process involving both the auditory and visual modalities. It is unknown whether and to what extent somatosensory stimulation to the facial skin modulates audio-visual speech perception. If speech perception is an embodied process, then somatosensory stimulation applied to the perceiver should influence audio-visual speech processing. Using the McGurk effect (the perceptual illusion that occurs when a sound is paired with the visual representation of a different sound, resulting in the perception of a third sound) we tested the prediction using a simple behavioral paradigm and at the neural level using event-related potentials (ERPs) and their cortical sources. We recorded ERPs from 64 scalp sites in response to congruent and incongruent audio-visual speech randomly presented with and without somatosensory stimulation associated with facial skin deformation. Subjects judged whether the production was /ba/ or not under all stimulus conditions. In the congruent audio-visual condition subjects identifying the sound as /ba/, but not in the incongruent condition consistent with the McGurk effect. Concurrent somatosensory stimulation improved the ability of participants to more correctly identify the production as /ba/ relative to the non-somatosensory condition in both congruent and incongruent conditions. ERP in response to the somatosensory stimulation for the incongruent condition reliably diverged 220 msec after stimulation onset. Cortical sources were estimated around the left anterior temporal gyrus, the right middle temporal gyrus, the right posterior superior temporal lobe and the right occipital region. The results demonstrate a clear multisensory convergence of somatosensory and audio-visual processing in both behavioral and neural processing consistent with the perspective that speech perception is a self-referenced, sensorimotor process.

Reading-Related Brain Changes in Audiovisual Processing: Cross-Sectional and Longitudinal MEG Evidence.

The ability to establish associations between visual objects and speech sounds is essential for human reading. Understanding the neural adjustments required for acquisition of these arbitrary audiovisual associations can shed light on fundamental reading mechanisms and help reveal how literacy builds on pre-existing brain circuits. To address these questions, the present longitudinal and cross-sectional MEG studies characterize the temporal and spatial neural correlates of audiovisual syllable congruency in children (age range, 4-9 years; 22 males and 20 females) learning to read. Both studies showed that during the first years of reading instruction children gradually set up audiovisual correspondences between letters and speech sounds, which can be detected within the first 400 ms of a bimodal presentation and recruit the superior portions of the left temporal cortex. These findings suggest that children progressively change the way they treat audiovisual syllables as a function of their reading experience. This reading-specific brain plasticity implies (partial) recruitment of pre-existing brain circuits for audiovisual analysis.SIGNIFICANCE STATEMENT Linking visual and auditory linguistic representations is the basis for the development of efficient reading, while dysfunctional audiovisual letter processing predicts future reading disorders. Our developmental MEG project included a longitudinal and a cross-sectional study; both studies showed that children's audiovisual brain circuits progressively change as a function of reading experience. They also revealed an exceptional degree of neuroplasticity in audiovisual neural networks, showing that as children develop literacy, the brain progressively adapts so as to better detect new correspondences between letters and speech sounds.

Audiovisual working memory and association with resting-state regional homogeneity

Multisensory processing is a prevalent research issue. However, multisensory working memory research has received inadequate attention. The present study aimed to investigate the behavioral performance of an audiovisual working memory task and its association with resting-state functional magnetic resonance imaging (fMRI) regional homogeneity (ReHo). A total of 128 healthy participants were enrolled in this study. The participants completed a modified Sternberg working memory task using complex auditory and visual objects as materials involved in different encoding conditions, including semantically congruent audiovisual, semantically incongruent audiovisual, and single modality of auditory or visual object encoding. Two subgroups received resting-state fMRI scans according to their behavioral performances. The results showed that the semantically congruent audiovisual object encoding sped up the later unisensory memory recognition in this task. Moreover, the high behavioral performance (response time, RT) group showed increased ReHo in the executive control network (ECN) and decreased ReHo in the default mode network (DMN) and saline network (SN). In addition, resting-state ReHo values in the ECN nodes (e.g., middle frontal gyrus and superior frontal gyrus) was correlated with RT. These findings suggested that semantically congruent audiovisual processing in working memory was superior to unisensory memory recognition and may be involved in the different functional networks such as ECN.

Anxiety biases audiovisual processing of social signals

In everyday life, information from multiple senses is integrated for a holistic understanding of emotion. Despite evidence of atypical multisensory perception in populations with socio-emotional difficulties (e.g., autistic individuals), little research to date has examined how anxiety impacts on multisensory emotion perception. Here we examined whether the level of trait anxiety in a sample of 56 healthy adults affected audiovisual processing of emotion for three types of stimuli: dynamic faces and voices, body motion and dialogues of two interacting agents, and circles and tones. Participants judged emotion from four types of displays – audio-only, visual-only, audiovisual congruent (e.g., angry face and angry voice) and audiovisual incongruent (e.g., angry face and happy voice) – as happy or angry, as quickly as possible. In one task, participants based their emotional judgements on information in one modality while ignoring information in the other, and in a second task they based their judgements on their overall impressions of the stimuli. The results showed that the higher trait anxiety group prioritized the processing of angry cues when combining faces and voices that portrayed conflicting emotions. Individuals in this group were also more likely to benefit from combining congruent face and voice cues when recognizing anger. The multisensory effects of anxiety were found to be independent of the effects of autistic traits. The observed effects of trait anxiety on multisensory processing of emotion may serve to maintain anxiety by increasing sensitivity to social-threat and thus contributing to interpersonal difficulties.

Audiovisual integration and the P2 component in adult Asperger’s syndrome: An ERP-study

BackgroundA widened temporal window of integration (TWI) in audiovisual processing has been detected for children with autism spectrum disorder. However, research indicates a narrowing of this TWI and an associated change in audiovisual integration in the course of development. MethodsTo further elucidate audiovisual integration processes in adulthood, we compared adult participants with Asperger’s Syndrome (AS) to healthy controls (HC) by using the sound-induced double flash illusion. For a better understanding of underlying neural mechanisms, event-related potentials were measured via electroencephalography (EEG). ResultsThe number of reported sound-induced flash illusions indicated audiovisual integration. A similar TWI size for both, participants with AS and HC, was found. Additionally, enhanced P2 amplitudes were detected for participants with AS compared to HC. ConclusionResults indicate an involvement of attentional processes in audiovisual perception in participants with AS.

The electrophysiology of audiovisual processing in visual narratives in adolescents with autism spectrum disorder

We investigated the semantic processing of the multimodal audiovisual combination of visual narratives with auditory descriptive words and auditory sounds in individuals with ASD. To this aim, we recorded ERPs to critical auditory words and sounds associated with events in visual narrative that were either semantically congruent or incongruent with the climactic visual event. A similar N400 effect was found both in adolescents with ASD and neurotypical adolescents (ages 9–16) when accessing different types of auditory information (i.e. words and sounds) into a visual narrative. This result might suggest that verbal information processing in ASD adolescents could be facilitated by direct association with meaningful visual information.In addition, we observed differences in scalp distribution of later brain responses between ASD and neurotypical adolescents. This finding might suggest ASD adolescents differ from neurotypical adolescents during the processing of the multimodal combination of visual narratives with auditory information at later stages of the process. In conclusion, the semantic processing of verbal information, typically impaired in individuals with ASD, can be facilitated when embedded into a meaningful visual information.

Graphomotor memory in Exner\u2019s area enhances word learning in the blind

Handwriting is thought to impede vocabulary learning in sighted adults because the motor execution of writing interferes with efficient audiovisual processing during encoding. However, the motor memory of writing may facilitate adult word learning when visual sensory inputs are severely restricted. Using functional MRI, we show that late-blind participants, but not sighted participants, learned novel words by recruiting the left dorsal premotor cortex known as Exner’s writing area and its functional coupling with the left hippocampus. During later recall, the phonological and semantic contents of these words are represented in the activation patterns of the left hippocampus as well as in those of left frontotemporal language areas. These findings suggest that motor codes of handwriting help blind participants maintain word-form representations during learning and retrieval. We propose that such reliance on the motor system reflects a broad architecture of the cerebral language network which encompasses the limb motor system as a hardwired component.

Audiovisual Speech Perception in Children with Autism Spectrum Disorders: Evidence from Visual Phonemic Restoration.

Children with autism spectrum disorders have been reported to be less influenced by a speaker's face during speech perception than those with typically development. To more closely examine these reported differences, a novel visual phonemic restoration paradigm was used to assess neural signatures (event-related potentials [ERPs]) of audiovisual processing in typically developing children and in children with autism spectrum disorder. Video of a speaker saying the syllable /ba/ was paired with (1) a synthesized /ba/ or (2) a synthesized syllable derived from /ba/ in which auditory cues for the consonant were substantially weakened, thereby sounding more like /a/. The auditory stimuli are easily discriminable; however, in the context of a visual /ba/, the auditory /a/ is typically perceived as /ba/, producing a visual phonemic restoration. Only children with ASD showed a large /ba/-/a/ discrimination response in the presence of a speaker producing /ba/, suggesting reduced influence of visual speech.

Audiovisual structural connectivity in musicians and non-musicians: a cortical thickness and diffusion tensor imaging study

Our sensory systems provide complementary information about the multimodal objects and events that are the target of perception in everyday life. Professional musicians’ specialization in the auditory domain is reflected in the morphology of their brains, which has distinctive characteristics, particularly in areas related to auditory and audio-motor activity. Here, we combined diffusion tensor imaging (DTI) with a behavioral measure of visually induced gain in pitch discrimination, and we used measures of cortical thickness (CT) correlations to assess how auditory specialization and musical expertise are reflected in the structural architecture of white and grey matter relevant to audiovisual processing. Across all participants (n = 45), we found a correlation (p < 0.001) between reliance on visual cues in pitch discrimination and the fractional anisotropy (FA) in the left inferior fronto-occipital fasciculus (IFOF), a structure connecting visual and auditory brain areas. Group analyses also revealed greater cortical thickness correlation between visual and auditory areas in non-musicians (n = 28) compared to musicians (n = 17), possibly reflecting musicians’ auditory specialization (FDR < 10%). Our results corroborate and expand current knowledge of functional specialization with a specific focus on audition, and highlight the fact that perception is essentially multimodal while uni-sensory processing is a specialized task.

Effects of stimulus intensity on audiovisual integration in aging across the temporal dynamics of processing

Previous studies have drawn different conclusions about whether older adults benefit more from audiovisual integration, and such conflicts may have been due to the stimulus features investigated in those studies, such as stimulus intensity. In the current study, using ERPs, we compared the effects of stimulus intensity on audiovisual integration between young adults and older adults. The results showed that inverse effectiveness, which depicts a phenomenon that lowing the effectiveness of sensory stimuli increases benefits of multisensory integration, was observed in young adults at earlier processing stages but was absent in older adults. Moreover, at the earlier processing stages (60–90 ms and 110–140 ms), older adults exhibited significantly greater audiovisual integration than young adults (all ps < 0.05). However, at the later processing stages (220–250 ms and 340–370 ms), young adults exhibited significantly greater audiovisual integration than old adults (all ps < 0.001). The results suggested that there is an age-related dissociation between early integration and late integration, which indicates that there are different audiovisual processing mechanisms in play between older adults and young adults.