Audiovisual Modalities Research Articles

The Visual Advantage Effect in Comparing Uni-Modal and Cross-Modal Probabilistic Category Learning.

People rely on multiple learning systems to complete weather prediction (WP) tasks with visual cues. However, how people perform in audio and audiovisual modalities remains elusive. The present research investigated how the cue modality influences performance in probabilistic category learning and conscious awareness about the category knowledge acquired. A modified weather prediction task was adopted, in which the cues included two dimensions from visual, auditory, or audiovisual modalities. The results of all three experiments revealed better performances in the visual modality relative to the audio and audiovisual modalities. Moreover, participants primarily acquired unconscious knowledge in the audio and audiovisual modalities, while conscious knowledge was acquired in the visual modality. Interestingly, factors such as the amount of training, the complexity of visual stimuli, and the number of objects to which the two cues belonged influenced the amount of conscious knowledge acquired but did not change the visual advantage effect. These findings suggest that individuals can learn probabilistic cues and category associations across different modalities, but a robust visual advantage persists. Specifically, visual associations can be learned more effectively, and are more likely to become conscious. The possible causes and implications of these effects are discussed.

Audiovisual integration in children with cochlear implants revealed through EEG and fNIRS

Sensory deprivation can offset the balance of audio versus visual information in multimodal processing. Such a phenomenon could persist for children born deaf, even after they receive cochlear implants (CIs), and could potentially explain why one modality is given priority over the other. Here, we recorded cortical responses to a single speaker uttering two syllables, presented in audio-only (A), visual-only (V), and audio-visual (AV) modes. Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) were successively recorded in seventy-five school-aged children. Twenty-five were children with normal hearing (NH) and fifty wore CIs, among whom 26 had relatively high language abilities (HL) comparable to those of NH children, while 24 others had low language abilities (LL). In EEG data, visual-evoked potentials were captured in occipital regions, in response to V and AV stimuli, and they were accentuated in the HL group compared to the LL group (the NH group being intermediate). Close to the vertex, auditory-evoked potentials were captured in response to A and AV stimuli and reflected a differential treatment of the two syllables but only in the NH group. None of the EEG metrics revealed any interaction between group and modality. In fNIRS data, each modality induced a corresponding activity in visual or auditory regions, but no group difference was observed in A, V, or AV stimulation. The present study did not reveal any sign of abnormal AV integration in children with CI. An efficient multimodal integrative network (at least for rudimentary speech materials) is clearly not a sufficient condition to exhibit good language and literacy.

Revisiting raters’ accent familiarity in speaking tests: Evidence that presentation mode interacts with accent familiarity to variably affect comprehensibility ratings

This controlled experimental study investigated the interaction of variables associated with rating the pronunciation component of high-stakes English-language-speaking tests such as IELTS and TOEFL iBT. One hundred experienced raters who were all either familiar or unfamiliar with Brazilian-accented English or Papua New Guinean Tok Pisin-accented English, respectively, were presented with speech samples in audio-only or audio-visual mode. Two-way ordinal regression with post hoc pairwise comparisons found that the presentation mode interacted significantly with accent familiarity to increase comprehensibility ratings (χ² = 88.005, df = 3, p < .0001), with presentation mode having a stronger effect in the interaction than accent familiarity (χ² = 59.328, df = 1, p < .0001). Based on odds ratios, raters were significantly more likely to score comprehensibility higher when the presentation mode was audio-visual (compared to audio-only) for both the unfamiliar (91% more likely) and familiar speakers (92.3% more likely). The results suggest that semi-direct speaking tests using audio-only or audio-visual modes of presentation should be evaluated through research to ascertain how accent familiarity and presentation mode interact to variably affect comprehensibility ratings. Such research may be beneficial to investigate the virtual modes of speaking test delivery that have emerged post-COVID-19.

Depression Recognition Using Remote Photoplethysmography From Facial Videos

Depression is a mental illness that may be harmful to an individual's health. The detection of mental health disorders in the early stages and a precise diagnosis are critical to avoid social, physiological, or psychological side effects. This work analyzes physiological signals to observe if different depressive states have a noticeable impact on the blood volume pulse (BVP) and the heart rate variability (HRV) response. Although typically, HRV features are calculated from biosignals obtained with contact-based sensors such as wearables, we propose instead a novel scheme that directly extracts them from facial videos, just based on visual information, removing the need for any contact-based device. Our solution is based on a pipeline that is able to extract complete remote photoplethysmography signals (rPPG) in a fully unsupervised manner. We use these rPPG signals to calculate over 60 statistical, geometrical, and physiological features that are further used to train several machine learning regressors to recognize different levels of depression. Experiments on two benchmark datasets indicate that this approach offers comparable results to other audiovisual modalities based on voice or facial expression, potentially complementing them. In addition, the results achieved for the proposed method show promising and solid performance that outperforms hand-engineered methods and is comparable to deep learning-based approaches.

Semi-Transparent, Pixel-Free Upconversion Goggles with Dual Audio-Visual Communication.

The intractable brittleness and opacity of the crystalline semiconductor restrict the prospect of developing low-cost imaging systems. Here, infrared visualization technologies are established with large-area, semi-transparent organic upconversion devices that bring high-resolution invisible images into sight without photolithography. To exploit all photoinduced charge carriers, a monolithic device structure is proposed built on the infrared-selective, single-component charge generation layer of chloroaluminum phthalocyanine (ClAlPc) coupled to two visible light-emitting layers manipulated with unipolar charges. Transient pump-probe spectroscopy reveals that the ClAlPc-based device exhibits an efficient charge dissociation process under forward bias. This process is indicated by the prompt and strong features of electroabsorption screening. Furthermore, by imposing the electric field, the ultrafast excited state dynamic suggests a prolonged charge carrier lifetime from the ClAlPc, which facilitates the charge utilization for upconversion luminance. For the first time, >30% of the infrared photons are utilized without photomultiplication strategies owing to the trivial spectrum overlap between ClAlPc and the emitter. In addition, the device can broadcast the acoustic signal by synchronizing the device frequency with the light source, which enables to operate it in dual audio-visual mode. The work demonstrates the potential of upconversion devices for affordable infrared imaging in wearable electronics.

Research on the design of interactive intelligent toy art experience based on multimodal sensory

Abstract Although many kinds of intelligent toys, interactive toys that can mobilize children’s sensory experience in many aspects and cultivate their creative thinking are still scarce. In this paper, interactive experience elements are built based on virtual reality technology, image recognition technology, human-computer interaction theory, and ergonomics theory. A brain interaction network based on multimodal sensory modulation is constructed to allow intelligent toys to perceive their surroundings effectively. And the dynamic causal model (DCM) is proposed to infer the connection strength and direction of neural activities among pre-defined cortical regions of the brain. The results showed that the classification accuracy of noisy audiovisual (AVd) modality in the BA18 region was 87.87% significantly higher than that of the STS region (81.41%). The average classification accuracy of the mono-visual noise-added (Vd) modality in the BA18 ROI was 79.40%, which was also significantly higher than that of the STS ROI at 65.12%. It indicates that the interactive design based on multimodal senses can effectively stimulate children’s brain interest areas, which makes children experience better and communicate more interactions. The research results of this paper provide new perspectives and have important implications for the application of interactive intelligence to toy art.

How are state telehealth policies associated with services offered by substance use disorder treatment facilities? Evidence from 2019 to 2022

BackgroundThe COVID-19 pandemic led several states to adopt policies permitting the delivery of substance use disorder treatment (SUDT) by telehealth. We assess the impact of state-level telehealth policies in 2020 that specifically permitted audio or audiovisual forms of telehealth offerings among SUDT facilities. ProcedureCross-sectional analysis of secondary data from between 2019 and 2022. Pre-pandemic, federal law permitted states to allow audiovisual telehealth modes for SUDT to a limited extent. 2020 laws permitted states to allow audio-only modes for the first time and strengthened ability to offer audiovisual modes. We compared national SUDT facility self-reported telehealth offerings in 2020 and beyond to 2019, in states that in 2020 had policies permitting audiovisual and audio only, compared to other states. Main findingsAmong outpatient SUDT facilities (n = 5227) present in all four years of our data, the proportion offering telehealth increased from 18% (n = 921) in 2019–26% in 2020, 60% in 2021, and 79% in 2022. We estimate an audiovisual and audio only policy in 2020 was associated with an increase in telehealth offering rates in 2022 of +16.5% points (pp) (95% CI [+10.4,+22.6]) compared to the rates in states with no such listed policy. There was little evidence of an influence on telehealth offering in 2020 (−2.9 pp, CI [−9.0,+3.2]) and 2021 (+0.6 pp, CI [−5.5,+6.7]). ConclusionsThe enactment of state-level telehealth policies that allow audio and audiovisual modalities may have increased SUDT facilities’ likelihood of offering telehealth services two years after enactment.

The Boys in the Band

Abstract This article analyzes the Spanish dubbed version and English-language version of Netflix’s recent adaptation of The Boys in the Band (Mantello 2020), which was initially a theater play by Crowley (1968), and compares it with the first English-language film adaptation (Friedkin 1970), also dubbed into Spanish. The concept of nostalgia is used here to analyze the resemiotization of different audiovisual modes (particularly cinematography) and how they are combined with the linguistic code to reimagine the narrative of the homosexual community in New York at the end of the 1960s. The English versions of the two films (released in 1970 and 2020) repeat the dialogues almost verbatim; however, the versions dubbed into Spanish offer different translation solutions. The time of production of both dubbed versions accounts for the viability of the expression of different sexual identities in the Spanish context at each moment. Based on this, the article reveals how the audiovisual modes of the latest version of The Boys in the Band, including its Spanish dubbing, contribute to nostalgic aesthetics and discursive representation of the emerging cultural and social gay American scene.

Co-creation and learning: an assessment of the pedagogical proposals in collaborative interactive non-fiction

Audiovisual co-creation has always had a pedagogical nature, which has been maintained in its adaptation to digital communication, although it is applied in different ways. This study analyses the pedagogical proposals in collaborative interactive non-fiction in the digital environment, and compares them with previous projects of participatory practices. As case studies we selected five interactive audiovisual projects that stand out for their educational intent and their preparation of teaching resources, and analyzed their pedagogical proposals. First, we looked at pedagogy in pre-digital or analogical collaborative creation, which was used to develop an original analytical table. We used this table to study the five collaborative projects through three identified categories. The results show that these categories continue in initiatives today, although the findings show a shift in the pedagogy about audiovisuals, particularly concerning the technical aspects, as well as a weakening of the connection between learning and participation in the collective production. Finally, we can highlight that combining educational practices with collaborative production on the Internet is important to increase citizen participation and give a social perspective to this audiovisual modality.

Behavioral Interventions on Periodontitis Patients to Improve Oral Hygiene: A Systematic Review.

The PubMed/MEDLINE, Cochrane Library, and Embase online databases were explored to identify relevant studies published before October 2022. Articles investigating the effects of different psychological approaches and intervention strategies on periodontitis patients' oral hygiene (OH) behavioral change were screened. 5460 articles were identified, and 21 fulfilled the inclusion criteria. In total, 2 studies tested audio-visual modalities, and the remaining 19 publications involved six psychological models of health-related behavioral interventions, including Social Cognitive Theory, the Theory of Planned Behavior, the Health Action Process Approach, Leventhal's self-regulatory theory, Motivational Interviewing, and Cognitive Behavioral Therapy. A meta-analysis of the results was not carried out due to the high heterogeneity among the interventions. Considering the limitations of the available studies, psychological interventions based on social cognitive models that combine some of the techniques of this model (goal setting, planning, self-monitoring, and feedback) may improve OH in periodontitis patients, having a positive impact on periodontal clinical outcomes. Delivering cognitive behavioral therapy in combination with motivational interviewing may result in an improvement in OH as evaluated by decreasing plaque and bleeding scores.

Audio-visual clear speech: Articulation, acoustics and perception of segments and tones

Research has established that clear speech with enhanced acoustic signal benefits segmental intelligibility. Less attention has been paid to visible articulatory correlates of clear-speech modifications, or to clear-speech effects at the suprasegmental level (e.g., lexical tone). Questions thus arise as to the extent to which clear-speech cues are beneficial in different input modalities and linguistic domains, and how different resources are incorporated. These questions address the fundamental argument in clear-speech research with respect to the trade-off between effects of signal-based phoneme-extrinsic modifications to strengthen overall acoustic salience versus code-based phoneme-specific modifications to maintain phonemic distinctions. In this talk, we report findings from our studies on audio-visual clear speech production and perception, including vowels and fricatives differing in auditory and visual saliency, and lexical tones believed to lack visual distinctiveness. In a 3-stream study, we use computer-vision techniques to extract visible facial cues associated with segmental and tonal productions in plain and clear speech, characterize distinctive acoustic features across speech styles, and compare audio-visual plain and clear speech perception. Findings are discussed in terms of how speakers and perceivers strike a balance between utilizing general saliency-enhancing and category-specific cues across audio-visual modalities and speech styles with the aim of improving intelligibility.

Spatial and temporal (non)binding of audiovisual rhythms in sensorimotor synchronisation

Human movement synchronisation with moving objects strongly relies on visual input. However, auditory information also plays an important role, since real environments are intrinsically multimodal. We used electroencephalography (EEG) frequency tagging to investigate the selective neural processing and integration of visual and auditory information during motor tracking and tested the effects of spatial and temporal congruency between audiovisual modalities. EEG was recorded while participants tracked with their index finger a red flickering (rate fV = 15 Hz) dot oscillating horizontally on a screen. The simultaneous auditory stimulus was modulated in pitch (rate fA = 32 Hz) and lateralised between left and right audio channels to induce perception of a periodic displacement of the sound source. Audiovisual congruency was manipulated in terms of space in Experiment 1 (no motion, same direction or opposite direction), and timing in Experiment 2 (no delay, medium delay or large delay). For both experiments, significant EEG responses were elicited at fV and fA tagging frequencies. It was also hypothesised that intermodulation products corresponding to the nonlinear integration of visual and auditory stimuli at frequencies fV ± fA would be elicited, due to audiovisual integration, especially in Congruent conditions. However, these components were not observed. Moreover, synchronisation and EEG results were not influenced by congruency manipulations, which invites further exploration of the conditions which may modulate audiovisual processing and the motor tracking of moving objects.

Integrating audio and visual modalities for multimodal personality trait recognition via hybrid deep learning.

Recently, personality trait recognition, which aims to identify people's first impression behavior data and analyze people's psychological characteristics, has been an interesting and active topic in psychology, affective neuroscience and artificial intelligence. To effectively take advantage of spatio-temporal cues in audio-visual modalities, this paper proposes a new method of multimodal personality trait recognition integrating audio-visual modalities based on a hybrid deep learning framework, which is comprised of convolutional neural networks (CNN), bi-directional long short-term memory network (Bi-LSTM), and the Transformer network. In particular, a pre-trained deep audio CNN model is used to learn high-level segment-level audio features. A pre-trained deep face CNN model is leveraged to separately learn high-level frame-level global scene features and local face features from each frame in dynamic video sequences. Then, these extracted deep audio-visual features are fed into a Bi-LSTM and a Transformer network to individually capture long-term temporal dependency, thereby producing the final global audio and visual features for downstream tasks. Finally, a linear regression method is employed to conduct the single audio-based and visual-based personality trait recognition tasks, followed by a decision-level fusion strategy used for producing the final Big-Five personality scores and interview scores. Experimental results on the public ChaLearn First Impression-V2 personality dataset show the effectiveness of our method, outperforming other used methods.

FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos

Deep learning based visual-to-sound generation systems have been developed that identify and create audio features from video signals. However, these techniques often fail to consider the time-synchronicity of the visual and audio features. In this paper we introduce a novel method for guiding a class-conditioned GAN to synthesize representative audio with temporally extracted visual information. We accomplish this visual-to-sound generation task by adapting the synchronicity traits between the audio-visual modalities. Our proposed FoleyGAN model is capable of conditioning action sequences of visual events leading to the generation of visually aligned realistic soundtracks. We expanded our previously proposed Automatic Foley data set. We evaluated FoleyGAN's synthesized sound output through human surveys that show noteworthy (on average 81%) audio-visual synchronicity performance. Our approach outperforms other baseline models and audio-visual data sets in statistical and ablation experiments achieving improved IS, FID and NDB scores. In ablation analysis we showed the significance of our visual and temporal feature extraction method as well as augmented performance of our generation network. Overall, our FoleyGAN model showed sound retrieval accuracy of 76.08% surpassing existing visual-to-audio synthesis deep neural networks.

The effect of visual speech information on linguistic release from masking.

Listeners often experience challenges understanding a person (target) in the presence of competing talkers (maskers). This difficulty reduces with the availability of visual speech information (VSI; lip movements, degree of mouth opening) and during linguistic release from masking (LRM; masking decreases with dissimilar language maskers). We investigate whether and how LRM occurs with VSI. We presented English targets with either Dutch or English maskers in audio-only and audiovisual conditions to 62 American English participants. The signal-to-noise ratio (SNR) was easy at 0 audio-only and -8 dB audiovisual in Experiment 1 and hard at -8 and -16 dB in Experiment 2 to assess the effects of modality on LRM across the same and different SNRs. We found LRM in the audiovisual condition for all SNRs and in audio-only for -8 dB, demonstrating reliable LRM for audiovisual conditions. Results also revealed that LRM is modulated by modality with larger LRM in audio-only indicating that introducing VSI weakens LRM. Furthermore, participants showed higher performance for Dutch maskers compared to English maskers with and without VSI. This establishes that listeners use both VSI and dissimilar language maskers to overcome masking. Our study shows that LRM persists in the audiovisual modality and its strength depends on the modality.

Preprocessing Challenges for Real World Affect Recognition

Real world human affect recognition requires immediate attention which is a significant aspect of humancomputer interaction. Audio-visual modalities can make a significant contribution by providing rich contextual information. Preprocessing is an important step in which the relevant information is extracted. It has a crucial impact on prominent feature extraction and further processing. The main aim is to highlight the challenges in preprocessing real world data. The research focuses on experimental testing and comparative analysis for preprocessing using OpenCV, Single Shot MultiBox Detector (SSD), DLib, Multi-Task Cascaded Convolutional Neural Networks (MTCNN), and RetinaFace detectors. The comparative analysis shows that MTCNN and RetinaFace give better performance in real world data. The performance of facial affect recognition using a pre-trained CNN model is analysed with a lab-controlled dataset CK+ and a representative wild dataset AFEW. This comparative analysis demonstrates the impact of preprocessing issues on feature engineering framework in real world affect recognition.

Does emotion recognition change across phases of the ovulatory cycle?

Recognizing emotions is an essential ability for successful interpersonal interaction. Prior research indicates some links between the endocrine system and emotion recognition ability, but only a few studies focused on within-subject differences across distinct ovulatory cycle phases and this ability. These studies have demonstrated mixed results that might be potentially due to heterogeneity in experimental tasks, methodologies, and lacking ecological validity. In the current study, we investigated associations between within-subject differences in ovarian hormones levels and emotion recognition from auditory, visual, and audiovisual modalities in N = 131 naturally cycling participants across the late follicular and mid-luteal phase of the ovulatory cycle. We applied a within-subject design with sessions in the late follicular and mid-luteal cycle phase, and also assessed salivary progesterone and estradiol in these sessions. Our findings did not reveal any significant difference in emotion recognition ability across two cycle phases. Thus, they emphasize the necessity of employing large-scale replication studies with well-established study designs along with proper statistical analyses. Moreover, our findings indicate that the potential link between ovulatory cycle phases (late follicular and mid-luteal) and emotion recognition ability might have been overestimated in previous studies, and may contribute to theoretical and practical implications of socio-cognitive neuroendocrinology.

Integrative interaction of emotional speech in audio-visual modality.

Emotional clues are always expressed in many ways in our daily life, and the emotional information we receive is often represented by multiple modalities. Successful social interactions require a combination of multisensory cues to accurately determine the emotion of others. The integration mechanism of multimodal emotional information has been widely investigated. Different brain activity measurement methods were used to determine the location of brain regions involved in the audio-visual integration of emotional information, mainly in the bilateral superior temporal regions. However, the methods adopted in these studies are relatively simple, and the materials of the study rarely contain speech information. The integration mechanism of emotional speech in the human brain still needs further examinations. In this paper, a functional magnetic resonance imaging (fMRI) study was conducted using event-related design to explore the audio-visual integration mechanism of emotional speech in the human brain by using dynamic facial expressions and emotional speech to express emotions of different valences. Representational similarity analysis (RSA) based on regions of interest (ROIs), whole brain searchlight analysis, modality conjunction analysis and supra-additive analysis were used to analyze and verify the role of relevant brain regions. Meanwhile, a weighted RSA method was used to evaluate the contributions of each candidate model in the best fitted model of ROIs. The results showed that only the left insula was detected by all methods, suggesting that the left insula played an important role in the audio-visual integration of emotional speech. Whole brain searchlight analysis, modality conjunction analysis and supra-additive analysis together revealed that the bilateral middle temporal gyrus (MTG), right inferior parietal lobule and bilateral precuneus might be involved in the audio-visual integration of emotional speech from other aspects.

Effect of Sensory Modality on Reaction Time in Individuals with Auditory Neuropathy Spectrum Disorder

PurposeTo investigate and compare the reaction time of individuals with auditory neuropathy in three modalities, auditory, visual, and audio-visual. The reaction time of individuals with auditory neuropathy was also compared with those with normal hearing. The relationship between reaction time across modalities and the duration of hearing loss in auditory neuropathy was also investigated. Methods and MaterialsThe reaction time of adults with auditory neuropathy and those with normal hearing was measured in the three modalities using the Choice reaction time task. ResultsThe auditory neuropathy group significantly had a longer reaction time than the normal hearing in all modalities. The trend of the mean reaction time differed across groups. Further, a significant difference in reaction time of the auditory neuropathy group was noted between auditory and visual mode, auditory and audio-visual mode. However, no significant difference between visual and audio-visual modalities was noted in reaction time. ConclusionSignificantly longer reaction time in auditory neuropathy is presumed to have resulted from neural conduction delay and impaired processing. The auditory neuropathy group can utilize visual cues for faster processing, and the study recommends an audio-visual mode for their management. In addition, the duration of hearing loss in auditory neuropathy had no relationship with reaction time across all modalities.

The different brain areas occupied for integrating information of hierarchical linguistic units: a study based on EEG and TMS.

Human language units are hierarchical, and reading acquisition involves integrating multisensory information (typically from auditory and visual modalities) to access meaning. However, it is unclear how the brain processes and integrates language information at different linguistic units (words, phrases, and sentences) provided simultaneously in auditory and visual modalities. To address the issue, we presented participants with sequences of short Chinese sentences through auditory, visual, or combined audio-visual modalities while electroencephalographic responses were recorded. With a frequency tagging approach, we analyzed the neural representations of basic linguistic units (i.e. characters/monosyllabic words) and higher-level linguistic structures (i.e. phrases and sentences) across the 3 modalities separately. We found that audio-visual integration occurs in all linguistic units, and the brain areas involved in the integration varied across different linguistic levels. In particular, the integration of sentences activated the local left prefrontal area. Therefore, we used continuous theta-burst stimulation to verify that the left prefrontal cortex plays a vital role in the audio-visual integration of sentence information. Our findings suggest the advantage of bimodal language comprehension at hierarchical stages in language-related information processing and provide evidence for the causal role of the left prefrontal regions in processing information of audio-visual sentences.