Audiovisual Stimuli Research Articles

Evaluating the effects of audiovisual delays on speech understanding with hearables

A key consideration in the development of real-time speech processing algorithms is latency between when speech is first captured and when the processed speech is transmitted. Much of the previous work informing the acceptability of processing latency has focused on the thresholds at which delays are detectable or annoying. This study aimed to explore the effects of delays on responding to a task necessitating speech understanding. Sentence-length audiovisual speech stimuli were modified such that the audio stream was delayed relative to the visual stream as well as to an attenuated copy of the audio stream by 10 and 400 ms, respectively. These stimuli were presented in conjunction with a pictorial referent-choice task to 50 normal-hearing participants. Participants responded slower to the task when the audio stream was delayed by 400 ms relative to the video stream. With an additional attenuated copy of the audio stream synchronized with the video stream, reaction time decreased, but this effect was limited to when the target word appeared near the onset of the sentence, highlighting the complexity of delay perception in the context of continuous speech. Implications for determining permissible processing delays for digital electronic hearing protection devices are discussed.

Cumulative multisensory discrepancies shape the ventriloquism aftereffect but not the ventriloquism bias.

Multisensory integration and recalibration are two processes by which perception deals with discrepant signals. Both are often studied in the spatial ventriloquism paradigm. There, integration is probed by the presentation of discrepant audio-visual stimuli, while recalibration manifests as an aftereffect in subsequent judgements of unisensory sounds. Both biases are typically quantified against the degree of audio-visual discrepancy, reflecting the possibility that both may arise from common underlying multisensory principles. We tested a specific prediction of this: that both processes should also scale similarly with the history of multisensory discrepancies, i.e. the sequence of discrepancies in several preceding audio-visual trials. Analyzing data from ten experiments with randomly varying spatial discrepancies we confirmed the expected dependency of each bias on the immediately presented discrepancy. And in line with the aftereffect being a cumulative process, this scaled with the discrepancies presented in at least three preceding audio-visual trials. However, the ventriloquism bias did not depend on this three-trial history of multisensory discrepancies and also did not depend on the aftereffect biases in previous trials - making these two multisensory processes experimentally dissociable. These findings support the notion that the ventriloquism bias and the aftereffect reflect distinct functions, with integration maintaining a stable percept by reducing immediate sensory discrepancies and recalibration maintaining an accurate percept by accounting for consistent discrepancies.

Mouth and facial informativeness norms for 2276 English words.

Mouth and facial movements are part and parcel of face-to-face communication. The primary way of assessing their role in speech perception has been by manipulating their presence (e.g., by blurring the area of a speaker's lips) or by looking at how informative different mouth patterns are for the corresponding phonemes (or visemes; e.g., /b/ is visually more salient than /g/). However, moving beyond informativeness of single phonemes is challenging due to coarticulation and language variations (to name just a few factors). Here, we present mouth and facial informativeness (MaFI) for words, i.e., how visually informative words are based on their corresponding mouth and facial movements. MaFI was quantified for 2276 English words, varying in length, frequency, and age of acquisition, using phonological distance between a word and participants' speechreading guesses. The results showed that MaFI norms capture well the dynamic nature of mouth and facial movements per word, with words containing phonemes with roundness and frontness features, as well as visemes characterized by lower lip tuck, lip rounding, and lip closure being visually more informative. We also showed that the more of these features there are in a word, the more informative it is based on mouth and facial movements. Finally, we demonstrated that the MaFI norms generalize across different variants of English language. The norms are freely accessible via Open Science Framework ( https://osf.io/mna8j/ ) and can benefit any language researcher using audiovisual stimuli (e.g., to control for the effect of speech-linked mouth and facial movements).

Difficulties in Treating Ethnic Minority Children with ADHD and the Role of Mobile Applications

There is proof that kids and teens from ethnic minorities get less health care than theirnon-minority peers, despite the fact that Attention Deficit Hyperactivity Disorder (ADHD) ismore common in these populations. Minority children might not receive timely, adequateADHD intervention because of the low socioeconomic position of minority parents, their lackof awareness regarding ADHD, and the available remedial techniques. People are looking tothe digital world for answers to challenges in their daily lives as the number of mobile applicationsgrows. People with impairments now have equal access to learning opportunitiesbecause to technological advancements. The majority of their time is now spent on mobiledevices by kids. As a result, using mobile applications to solve their problems would be quitesuccessful. Mobile learning, also known as e-learning programs that utilize mobile devices,can be employed as a form of therapy to enhance both executive functioning and generalwell-being. Also, by practicing cognitive and metacognitive task abilities more frequently, theyincrease motivation in kids and teenagers. Also, the understanding and retention of the presentedinformation might be aided by the audiovisual stimuli offered through smartphoneapplications. They may therefore serve as crucial learning aids for kids with ADHD. The currentstudy examines the function and efficacy of mobile applications in enhancing the qualityof life of children from ethnic minorities who have Attention Deficit Hyperactivity Disorder(ADHD), as well as the barriers that may impede or delay their access to treatment services.

Metacognitive awareness in the sound-induced flash illusion.

Hundreds (if not thousands) of multisensory studies provide evidence that the human brain can integrate temporally and spatially discrepant stimuli from distinct modalities into a singular event. This process of multisensory integration is usually portrayed in the scientific literature as contributing to our integrated, coherent perceptual reality. However, missing from this account is an answer to a simple question: how do confidence judgements compare between multisensory information that is integrated across multiple sources, and multisensory information that comes from a single, congruent source in the environment? In this paper, we use the sound-induced flash illusion to investigate if confidence judgements are similar across multisensory conditions when the numbers of auditory and visual events are the same, and the numbers of auditory and visual events are different. Results showed that congruent audiovisual stimuli produced higher confidence than incongruent audiovisual stimuli, even when the perceptual report was matched across the two conditions. Integrating these behavioural findings with recent neuroimaging and theoretical work, we discuss the role that prefrontal cortex may play in metacognition, multisensory causal inference and sensory source monitoring in general. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Neural basis of sound-symbolic pseudoword-shape correspondences

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like ‘mohloh’ and ‘kehteh’, are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants (n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword (‘mohloh’ or ‘kehteh’) and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.

Why is it difficult for children and adults to follow a person's eye gaze in polynomial social relationships with compound audio-visual stimuli: An eye-tracking study.

Acquiring triadic social interactions could facilitate language and communication skills in early infancy. However, studies have rarely investigated polynomial relationships, defined as relationships among the self, two or more people, and objects. During the development from a child to an adult, the responsiveness to a preferred stimulus modality changes from visual to auditory dominance. Nevertheless, how people observe compound visual stimuli in polynomial social relationships and why it is difficult to ignore auditory cues remain unclear. Moreover, there is a need to identify differences between children's and adults' observing latencies in the time to the first fixation when detecting a stimulus. This study examined whether participants (24 adults and 19 children) demonstrated similar gaze patterns under triadic and polyadic conditions. The participants observed a target visual stimulus looked at by a face stimulus while we presented spoken names, either congruent or incongruent with the target visual stimulus. The results indicated that when the number of people in social relationships increased, children and adults decreased fixations on the target face and the stimulus and showed a shorter mean fixation duration on the face. Moreover, children had longer latencies and more fixation errors for the target stimulus, which might reflect children's difficulties in communicating with others. We expect that understanding children's communication transition from triadic to polynomial social relationships with audio-visual stimulus congruencies would facilitate understanding language development and social communication patterns.

Dynamic audio-visual correspondence in musicians and non-musicians

Most previous audio-visual crossmodal correspondence studies focused on static visual or auditory stimuli. Moreover, some researchers have found that music training can effectively improve the interaction between auditory and visual channels by enhancing neural plasticity. This study focused on whether crossmodal correspondence occurs when people face dynamic visual and auditory stimuli, and whether it is affected by musical training. Participants were asked to judge different changes in pitch (rise and drop) by showing them different patterns of visual circle motion (enlarged, reduced, and unchanged). The results revealed that the audio-visual congruent combinations (the pitch rise when the circle was enlarged and the pitch drop when the circle was reduced) significantly shortened participants’ response times, whereas the audio-visual neutral combinations (the unchanged shape) had the highest accuracy. Participants with musical training were faster than participants without musical training to judge pitch changes in conditions where the association between pitch and shape size was incongruent. These findings provide empirical evidence for dynamic audio-visual crossmodal correspondence and shed light on the bright prospect of using congruent audio-visual stimuli in animation.

The effect of background audio and audiovisual stimuli on students' autonomic responses during and after an experimental academic examination.

Due to the Covid-19 pandemic lockdown during the online-distant education period, certain students tended to combine their courses and homework with TV or social media news or other media content, such as classical music, including a wealth of audio and audiovisual stimuli. As the audio and audiovisual stimuli existing in a learning environment may affect students' autonomic nervous system (ANS) responses negatively, the present study aimed to monitor the impact of background TV, classical music, and silence on students' ANS activity represented by heart rate (HR), heart rate variability (HRV), blood volume amplitude (BVA), and skin conductance level (SCL) during and after an experimental academic examination. Seventy-six students were randomly allocated to background TV, classical music, or silence groups. The experiment with repeated measures design consisted of four consecutive periods: baseline, anticipation, challenge, and recovery, lasting 4 min each. Within-subject analyses indicated significant HRV decrement only in the background TV group. Regardless of the experimental groups, HR and SCL increased while BVA decreased during the task. In addition, the between-subject analysis showed that the background TV group experienced significantly larger changes in HR and HRV parameters compared to the other experimental groups relative to their respective baseline measurements. Based on these results, we concluded that relative to classical music and silence, background TV, including audiovisual and verbal stimuli, extant in a learning environment might raise students' sympathetic activity. Further, classical music, without lyrics, may suppress the withdrawal of vagal activity and elevate the autonomic regulation capacity during the academic reading comprehension task. HRV is a more valid and reliable indicator of students' autonomic responses during a challenging academic task.

MultiTab: A Novel Portable Device to Evaluate Multisensory Skills.

To infer spatial-temporal features of an external event we are guided by multisensory cues, with intensive research showing an enhancement in the perception when information coming from different sensory modalities are integrated. In this scenario, the motor system seems to also have an important role in boosting perception. With the present work, we introduce and validate a novel portable technology, named MultiTab, which is able to provide auditory and visual stimulation, as well as to measure the user's manual responses. Our preliminary results indicate that MultiTab reliably induces multisensory integration in a spatial localization task, shown by significantly reduced manual response times in the localization of audiovisual stimuli compared to unisensory stimuliClinical relevance- The current work presents a novel portable device that could contribute to the clinical evaluation of multisensory processing as well as spatial perception. In addition, by promoting and recording manual actions, MultiTab could be especially suitable for the design of rehabilitative protocols using multisensory motor training.

The impact of cardiac phases on multisensory integration

The brain continuously processes information coming from both the external environment and visceral signals generated by the body. This constant information exchange between the body and the brain allows signals originating from the oscillatory activity of the heart, among others, to influence perception. Here, we investigated how the cardiac phase modulates multisensory integration, which is the process that allows information from multiple senses to combine non-linearly to reduce environmental uncertainty. Forty healthy participants completed a Simple Detection Task with unimodal (Auditory, Visual, Tactile) and bimodal (Audio-Tactile, Audio-Visual, Visuo-Tactile) stimuli presented 250 ms and 500 ms after the R-peak of the electrocardiogram, that is, systole and diastole, respectively. First, we found a nonspecific effect of the cardiac cycle phases on detection of both unimodal and bimodal stimuli. Reaction times were faster for stimuli presented during diastole, compared to systole. Then, applying the Race Model Inequality approach to quantify multisensory integration, Audio-Tactile and Visuo-Tactile, but not Audio-Visual stimuli, showed higher integration when presented during diastole than during systole. These findings indicate that the impact of the cardiac phase on multisensory integration may be specific for stimuli including somatosensory (i.e., tactile) inputs. This suggests that the heartbeat-related noise, which according to the interoceptive predictive coding theory suppresses somatosensory inputs, also affects multisensory integration during systole. In conclusion, our data extend the interoceptive predictive coding theory to the multisensory domain. From a more mechanistic view, they may reflect a reduced optimization of neural oscillations orchestrating multisensory integration during systole.

First steps into the pupillometry multiverse of developmental science

Pupillometry has been widely implemented to investigate cognitive functioning since infancy. Like most psychophysiological and behavioral measures, it implies hierarchical levels of arbitrariness in preprocessing before statistical data analysis. By means of an illustrative example, we checked the robustness of the results of a familiarization procedure that compared the impact of audiovisual and visual stimuli in 12-month-olds. We adopted a multiverse approach to pupillometry data analysis to explore the role of (1) the preprocessing phase, that is, handling of extreme values, selection of the areas of interest, management of blinks, baseline correction, participant inclusion/exclusion and (2) the modeling structure, that is, the incorporation of smoothers, fixed and random effects structure, in guiding the parameter estimation. The multiverse of analyses shows how the preprocessing steps influenced the regression results, and when visual stimuli plausibly predicted an increase of resource allocation compared with audiovisual stimuli. Importantly, smoothing time in statistical models increased the plausibility of the results compared to those nested models that do not weigh the impact of time. Finally, we share theoretical and methodological tools to move the first steps into (rather than being afraid of) the inherent uncertainty of infant pupillometry.

Cardiac interoception in infants: Behavioral and neurophysiological measures in various emotional and self-related contexts.

Interoception, the perception of internal bodily signals, is fundamental to our sense of self. Even though theoretical accounts suggest an important role for interoception in the development of the self, empirical investigations are limited, particularly in infancy. Previous studies used preferential-looking paradigms to assess the detection of sensorimotor and multisensory contingencies in infancy, usually related to proprioception and touch. So far, only one recent study reported that infants discriminated between audiovisual stimuli presented synchronously or asynchronously with their heartbeat. This discrimination was related to the amplitude of the infant's heartbeat evoked potentials (HEP), a neural correlate of interoception. In the current study, we measured looking preferences between synchronous and asynchronous visuocardiac (bimodal), and audiovisuocardiac (trimodal) stimuli as well as the HEP in conditions of different emotional contexts and with different degrees of self-relatedness in a mirror-like setup. While the infants preferred trimodal to bimodal stimuli, we did not observe the predicted differences between synchronous and asynchronous stimulation. Furthermore, the HEP was not modulated by emotional context or self-relatedness. These findings do not support previously published results and highlight the need for further studies on the early development of interoception in relation to the development of the self.

Change of rapid temporal recalibration magnitude for audiovisual asynchrony with modulation of temporal binding window width: A preliminary investigation.

The subjective synchrony perception for audiovisual stimuli is affected by previous temporal information. The point of subjective simultaneity is shifted toward the same asynchronous direction of audiovisual stimuli in a previous trial. This phenomenon is called "rapid temporal recalibration." The factors that modulate the magnitude of rapid temporal recalibration have not been fully investigated. Previously, a positive correlation has been found between the magnitude of rapid temporal recalibration and the width of the temporal binding window (TBW). This preliminary study examined the causal relationship between TBW size and rapid recalibration magnitude using a single experimental group comparison design. In this experiment, the magnitude of rapid recalibration was compared before and after perceptual training, which narrowed the TBW width. The results indicated that the magnitude of rapid recalibration was reduced by perceptual training. Therefore, it was speculated that TBW size determined the magnitude of rapid recalibration. This causal relationship helps elucidate the mechanisms of the adaptation for temporal lags between visual and auditory sensations.

Multimodal Temporal Pattern Discrimination Is Encoded in Visual Cortical Dynamics.

Discriminating between temporal features in sensory stimuli is critical to complex behavior and decision-making. However, how sensory cortical circuit mechanisms contribute to discrimination between subsecond temporal components in sensory events is unclear. To elucidate the mechanistic underpinnings of timing in primary visual cortex (V1), we recorded from V1 using two-photon calcium imaging in awake-behaving mice performing a go/no-go discrimination timing task, which was composed of patterns of subsecond audiovisual stimuli. In both conditions, activity during the early stimulus period was temporally coordinated with the preferred stimulus. However, while network activity increased in the preferred condition, network activity was increasingly suppressed in the nonpreferred condition over the stimulus period. Multiple levels of analyses suggest that discrimination between subsecond intervals that are contained in rhythmic patterns can be accomplished by local neural dynamics in V1.

Higher synchronization stability with piano experience: relationship with finger and presentation modality

BackgroundSynchronous finger tapping to external sensory stimuli is more stable for audiovisual combined stimuli than sole auditory or visual stimuli. In addition, piano players are superior in synchronous tapping and manipulating the ring and little fingers as compared to inexperienced individuals. However, it is currently unknown whether the ability to synchronize to external sensory stimuli with the ring finger is at the level of the index finger in piano players. The aim of this study was to compare the effect of piano experience on synchronization stability between the index and ring fingers using auditory, visual, and audiovisual combined stimuli.MethodsThirteen piano players and thirteen novices participated in this study. They were instructed to tap with their index or ring finger synchronously to auditory, visual, and audiovisual combined stimuli. The stimuli were presented from an electronic metronome at 1 Hz, and the tapping was performed 30 times in each condition. We analyzed standard deviation of intervals between the stimulus onset and the tap onset as synchronization stability.ResultsSynchronization stability for visual stimuli was lower during ring than index finger tapping in novices; however, this decline was absent in piano players. Also, piano players showed the higher synchronization stability for audiovisual combined stimuli than sole visual and auditory stimuli when tapping with the index finger. On the other hand, in novices, synchronization stability was higher for audiovisual combined stimuli than only visual stimuli.ConclusionsThese findings suggest that improvements of both sensorimotor processing and finger motor control by piano practice would contribute to superior synchronization stability.

Visual Motor Reaction Times Predict Receptive and Expressive Language Development in Early School-Age Children.

Proficiency of multisensory processing and motor skill are often associated with early cognitive, social, and language development. However, little research exists regarding the relationship between multisensory motor reaction times (MRTs) to auditory, visual and audiovisual stimuli, and classical measures of receptive language and expressive vocabulary development in school-age children. Thus, this study aimed to examine the concurrent development of performance in classical tests of receptive (Peabody Picture Vocabulary Test; abbreviated as PPVT) and expressive vocabulary (Expressive Vocabulary Test; abbreviated as EVT), nonverbal intelligence (NVIQ) (determined with the aid of Raven's Colored Progressive Matrices; abbreviated as RCPM), speed of visual-verbal processing in the Rapid Automatic Naming (RAN) test, Eye-Hand Co-ordination (EHC) in the SLURP task, and multisensory MRTs, in children (n = 75), aged between 5 and 10 years. Bayesian statistical analysis showed evidence for age group differences in EVT performance, while PPVT was only different for the youngest group of children aged 5-6, supporting different developmental trajectories in vocabulary acquisition. Bayesian correlations revealed evidence for associations between age, NVIQ, and vocabulary measures, with decisive evidence and a higher correlation (r = 0.57 to 0.68) between EVT, MRT tasks, and EHC visuomotor processing. This was further supported by regression analyses indicating that EVT performance was the strongest unique predictor of multisensory MRTs, EHC, and RAN time. Additionally, visual MRTs were found to predict both receptive and expressive vocabulary. The findings of the study have important implications as accessible school-based assessments of the concurrent development of NVIQ, language, and multisensory processing; and hence as rapid and timely measures of developmental and neurodevelopmental status.

Functional MRI responses to naturalistic stimuli are increasingly typical across early childhood

While findings show that throughout development, there are child- and age-specific patterns of brain functioning, there is also evidence for significantly greater inter-individual response variability in young children relative to adults. It is currently unclear whether this increase in functional “typicality” (i.e., inter-individual similarity) is a developmental process that occurs across early childhood, and what changes in BOLD response may be driving changes in typicality. We collected fMRI data from 81 typically developing 4–8-year-old children during passive viewing of age-appropriate television clips and asked whether there is increasing typicality of brain response across this age range. We found that the “increasing typicality” hypothesis was supported across many regions engaged by passive viewing. Post hoc analyses showed that in a priori ROIs related to language and face processing, the strength of the group-average shared component of activity increased with age, with no concomitant decline in residual signal or change in spatial extent or variability. Together, this suggests that increasing inter-individual similarity of functional responses to audiovisual stimuli is an important feature of early childhood functional brain development.

Audio-visual speech perception of plosive consonants by CG learners of English

Second language (L2) speech perception can be a challenging process, as listeners have to cope with imperfect auditory signals and imperfect L2 knowledge. However, the aim of L2 speech perception is to extract linguistic meaning and enable communication between interlocutors in the language of input. Normal-hearing listeners can effortlessly perceive and understand the auditory message(s) conveyed, regardless of distortions and background noise, as they can endure a dramatic decrease in the amount of spectral and temporal information present in the auditory signal. In their attempt to recognise speech, listeners can be substantially assisted by looking at the face of the speaker. Visual perception is important even in the case of intelligible speech sounds, indicating that auditory and visual information should be combined. The present study examines how audio-visual integration affects Cypriot-Greek (CG) listeners’ recognition performance of plosive consonants on word-level in L2 English. The participants were 14 first language (L1) CG users, who were non-native speakers of English. They completed a perceptual minimal-set task requiring the extraction of speech information from unimodal auditory stimuli, unimodal visual stimuli, bimodal audio-visual congruent stimuli, and incongruent stimuli. The findings indicated that overall performance was better in the bimodal congruent task. The results point to the multisensory speech-specific mode of perception, which plays an important role in alleviating the majority of moderate to severe L2 comprehension difficulties. CG listeners’ success seems to depend upon the ability to relate what they see to what they hear.

What Makes the Detection of Movement Different Within the Autistic Traits Spectrum? Evidence From the Audiovisual Depth Paradigm.

Atypical sensory processing is now considered a diagnostic feature of autism. Although multisensory integration (MSI) may have cascading effects on the development of higher-level skills such as socio-communicative functioning, there is a clear lack of understanding of how autistic individuals integrate multiple sensory inputs. Multisensory dynamic information is a more ecological construct than static stimuli, reflecting naturalistic sensory experiences given that our environment involves moving stimulation of more than one sensory modality at a time. In particular, depth movement informs about crucial social (approaching to interact) and non-social (avoiding threats/collisions) information. As autistic characteristics are distributed on a spectrum over clinical and general populations, our work aimed to explore the multisensory integration of depth cues in the autistic personality spectrum, using a go/no-go detection task. The autistic profile of 38 participants from the general population was assessed using questionnaires extensively used in the literature. Participants performed a detection task of auditory and/or visual depth moving stimuli compared to static stimuli. We found that subjects with high-autistic traits overreacted to depth movement and exhibited faster reaction times to audiovisual cues, particularly when the audiovisual stimuli were looming and/or were presented at a fast speed. These results provide evidence of sensory particularities in people with high-autistic traits and suggest that low-level stages of multisensory integration could operate differently all along the autistic personality spectrum.