Audio Modalities Research Articles

Investigating the effect of multimodality and sentiments on speaking assessments: a facial emotional analysis.

This quasi-experimental study aimed to determine the relationship between (i) oral language ability and emotions represented by facial emotions, and (ii) modality of assessment (audios versus videos) and sentiments embedded in each modality. Sixty university students watched and/or listened to four selected audio-visual stimuli and orally answered follow-up comprehension questions. One stimulus was designed to evoke happiness while the other, sadness. Participants' facial emotions during the answering were measured using the FaceReader technology. In addition, four trained raters assessed the responses of the participants. An analysis of the FaceReader data showed that there were significant main and interaction effects of sentiment and modality on participants' facial emotional expression. Notably, there was a significant difference in the amount of facial emotions evoked by (i) the happy vs. sad sentiment videos and (ii) video vs. audio modalities. In contrast, sentiments embedded in the stimuli and modalities had no significant effect on the measured speaking performance of the participants. Nevertheless, we found a number of significant correlations between the participants' test scores and some of their facial emotions evoked by the stimuli. Implications of these findings for the assessment of oral communication are discussed.

Analytical review of methods for automatic detection of user engagement in virtual communication

Introduction: The solution of the task of the recognition and assessment of user engagement in the acts of human-machine interaction or telecommunication, achieved through the use of automatic means, is highly important in computer recognition of human psycho-emotional states. This is necessary for e-learning, business and entertainment applications design. Purpose: To conduct a comparative analysis of the current information support in the field of automatic recognition and assessment of user involvement in human-machine interaction or virtual communication, as well as to establish a methodology for building a data body based on the idea of multimodal communication. Results: The conducted analysis of research papers has shown that in most existing databases there is a substantial lack of data for natural online communication. Moreover, not all databases contain different modalities in “human-machine-human” communication system. Text and audio modalities turn out to be important for a multilevel engagement classification task, aimed at the determination of engagement intensity. It is also promising to take into account “body language” features, such as facial expressions, movements of the body and the head, gestures. For the correct assessment of involvement, an engagement database must contain meta-data on the psycho-emotional states of communicants. Neural network-based approaches to the automatic detection of user engagement show the best performance. Practical relevance: Based on the obtained analytical conclusions, the authors of the paper are going to elaborate an original software system for automatic recognition of user engagement, and to collect a data set for machine learning purposes. The presented review formulates basic requirements for such systems and contributes to the solution of the problem of automatic recognition of psycho-emotional states. Discussion: The survey leads to the conclusion that the notion of engagement as understood in studies on automatic emotion recognition differs from that used in psychology. User (or communicant) engagement in terms of info- and communicative sphere implies the manifestation of a person's mental activity level (emotional, cognitive, and behavioral components) changing dynamically while interacting with another person or computer system.

An autoencoder-based feature level fusion for speech emotion recognition

Although speech emotion recognition is challenging, it has broad application prospects in human-computer interaction. Building a system that can accurately and stably recognize emotions from human languages can provide a better user experience. However, the current unimodal emotion feature representations are not distinctive enough to accomplish the recognition, and they do not effectively simulate the inter-modality dynamics in speech emotion recognition tasks. This paper proposes a multimodal method that utilizes both audio and semantic content for speech emotion recognition. The proposed method consists of three parts: two high-level feature extractors for text and audio modalities, and an autoencoder-based feature fusion. For audio modality, we propose a structure called Temporal Global Feature Extractor (TGFE) to extract the high-level features of the time-frequency domain relationship from the original speech signal. Considering that text lacks frequency information, we use only a Bidirectional Long Short-Term Memory network (BLSTM) and attention mechanism to simulate an intra-modal dynamic. Once these steps have been accomplished, the high-level text and audio features are sent to the autoencoder in parallel to learn their shared representation for final emotion classification. We conducted extensive experiments on three public benchmark datasets to evaluate our method. The results on Interactive Emotional Motion Capture (IEMOCAP) and Multimodal EmotionLines Dataset (MELD) outperform the existing method. Additionally, the results of CMU Multi-modal Opinion-level Sentiment Intensity (CMU-MOSI) are competitive. Furthermore, experimental results show that compared to unimodal information and autoencoder-based feature level fusion, the joint multimodal information (audio and text) improves the overall performance and can achieve greater accuracy than simple feature concatenation.

Audio-video fusion strategies for active speaker detection in meetings

Meetings are a common activity in professional contexts, and it remains challenging to endow vocal assistants with advanced functionalities to facilitate meeting management. In this context, a task like active speaker detection can provide useful insights to model interaction between meeting participants. Detection of the active speaker can be performed using only video based on the movements of the participants of a meeting. Depending on the assistant design and each participant position regarding the device, active speaker detection can benefit from information coming from visual and audio modalities. Motivated by our application context related to advanced meeting assistant, we want to combine audio and visual information to achieve the best possible performance. In this paper, we propose two different types of fusion (naive fusion and attention-based fusion) for the detection of the active speaker, combining two visual modalities and an audio modality through neural networks. In addition, the audio modality is mainly processed using neural networks. For comparison purpose, classical unsupervised approaches for audio feature extraction are also used. We expect visual data centered on the face of each participant to be very appropriate for detecting voice activity, based on the detection of lip and facial gestures. Thus, our baseline system uses visual data (video) and we chose a 3D Convolutional Neural Network (CNN) architecture, which is effective for simultaneously encoding appearance and movement. To improve this system, we supplemented the visual information by processing the audio stream with a CNN or an unsupervised speaker diarization system. We have further improved this system by adding visual modality information using motion through optical flow. We evaluated our proposal with a public and state-of-the-art benchmark: the AMI corpus. We analysed the contribution of each system to the merger carried out in order to determine if a given participant is currently speaking. We also discussed the results we obtained. Besides, we have shown that, for our application context, adding motion information greatly improves performance. Finally, we have shown that attention-based fusion improves performance while reducing the standard deviation.

Audio–visual collaborative representation learning for Dynamic Saliency Prediction

The Dynamic Saliency Prediction (DSP) task simulates the human selective attention mechanism to perceive a dynamic scene, which is significant and imperative in many vision tasks. Most of existing methods only consider visual cues, while neglecting the accompanying audio information, which can provide complementary information for the scene understanding. Indeed, there exists a strong relation between auditory and visual cues, and humans generally perceive the surrounding scene by collaboratively sensing these cues. Motivated by this, an audio–visual collaborative representation learning method is proposed for the DSP task, which explores the implicit knowledge in the audio modality to better predict the dynamic saliency map by assisting the visual modality. The proposed method consists of three parts: (1) audio–visual encoding, (2) audio–visual localization, and (3) collaborative integration parts. First, a refined SoundNet architecture is adopted to encode audio modality for obtaining corresponding features, and a modified 3D ResNet-50 architecture is employed to learn visual features, containing both spatial location and temporal motion information. Secondly, an audio–visual localization part is devised to locate the sounding salient object in the visual scene by learning the correspondence between audio and visual information. Thirdly, a collaborative integration part is devised to adaptively aggregate the audio–visual information and a center-bias prior to generate the final saliency map. Extensive experiments are conducted on six challenging audiovisual eye-tracking datasets, namely DIEM, AVAD, Coutrot1, Coutrot2, SumMe, and ETMD, which shows significant superiority over state-of-the-art DSP models.

Multimodal Emotional Classification Based on Meaningful Learning

Emotion recognition has become one of the most researched subjects in the scientific community, especially in the human–computer interface field. Decades of scientific research have been conducted on unimodal emotion analysis, whereas recent contributions concentrate on multimodal emotion recognition. These efforts have achieved great success in terms of accuracy in diverse areas of Deep Learning applications. To achieve better performance for multimodal emotion recognition systems, we exploit Meaningful Neural Network Effectiveness to enable emotion prediction during a conversation. Using the text and the audio modalities, we proposed feature extraction methods based on Deep Learning. Then, the bimodal modality that is created following the fusion of the text and audio features is used. The feature vectors from these three modalities are assigned to feed a Meaningful Neural Network to separately learn each characteristic. Its architecture consists of a set of neurons for each component of the input vector before combining them all together in the last layer. Our model was evaluated on a multimodal and multiparty dataset for emotion recognition in conversation MELD. The proposed approach reached an accuracy of 86.69%, which significantly outperforms all current multimodal systems. To sum up, several evaluation techniques applied to our work demonstrate the robustness and superiority of our model over other state-of-the-art MELD models.

Human motion intention recognition method with visual, audio, and surface electromyography modalities for a mechanical hand in different environments

Mechanical hand is widely used in industrial and rehabilitation fields. Good interaction will enable it to execute human ideas quickly and complete related tasks with humans well. Different environments will affect its performance. Therefore, the modality diversity and environmental adaptability of human–machine interaction are important.In this study, a visual modality was developed based on Mediapipe framework and Tensorflow lite. An audio modality was developed based on Chinese finger-guessing game terms and non-specific voice recognition technology. A surface electromyography (sEMG) modality was developed based on machine learning, and a touch interface was developed based on serial touch screen. Furthermore, the user can choose different modalities according to the environment through the touch screen.The experimental results show that the average accuracies of visual modality, audio modality, and sEMG modality are 98.2463%±1.5057%, 97.3132%±0.692%, and 96.3454%±2.0108%, respectively. On a computer equipped with Intel (R) Core (TM) i5-1137G7 CPU, the execution time of visual modality, audio modality, and sEMG modality for a single action in Python 3.6 are 0.03 s, 0.81 s, and 0.19 s, respectively.Compared with several existing methods, the proposed method has rich modalities, better accuracy, well real-time performance of motion recognition, and real-time modality switching function.

Joint modelling of audio-visual cues using attention mechanisms for emotion recognition

Emotions play a crucial role in human-human communications with complex socio-psychological nature. In order to enhance emotion communication in human-computer interaction, this paper studies emotion recognition from audio and visual signals in video clips, utilizing facial expressions and vocal utterances. Thereby, the study aims to exploit temporal information of audio-visual cues and detect their informative time segments. Attention mechanisms are used to exploit the importance of each modality over time. We propose a novel framework that consists of bi-modal time windows spanning short video clips labeled with discrete emotions. The framework employs two networks, with each one being dedicated to one modality. As input to a modality-specific network, we consider a time-dependent signal deriving from the embeddings of the video and audio modalities. We employ the encoder part of the Transformer on the visual embeddings and another one on the audio embeddings. The research in this paper introduces detailed studies and meta-analysis findings, linking the outputs of our proposition to research from psychology. Specifically, it presents a framework to understand underlying principles of emotion recognition as functions of three separate setups in terms of modalities: audio only, video only, and the fusion of audio and video. Experimental results on two datasets show that the proposed framework achieves improved accuracy in emotion recognition, compared to state-of-the-art techniques and baseline methods not using attention mechanisms. The proposed method improves the results over baseline methods by at least 5.4%. Our experiments show that attention mechanisms reduce the gap between the entropies of unimodal predictions, which increases the bimodal predictions’ certainty and, therefore, improves the bimodal recognition rates. Furthermore, evaluations with noisy data in different scenarios are presented during the training and testing processes to check the framework’s consistency and the attention mechanism’s behavior. The results demonstrate that attention mechanisms increase the framework’s robustness when exposed to similar conditions during the training and the testing phases. Finally, we present comprehensive evaluations of emotion recognition as a function of time. The study shows that the middle time segments of a video clip are essential in the case of using audio modality. However, in the case of video modality, the importance of time windows is distributed equally.

The limited effects of visual and audio modalities on second language listening comprehension

This study examined the effects of watching gestures and lip movement on second language (English) listening comprehension. The participants were 30 high and 22 low proficient learners of English. There were six listening tasks combining two variables: modality and difficulty. The modality consisted of three types of assessments: Body (the upper half of body was visible), Face (close up view), and Audio. The difficulty consisted of two levels: Easy and Hard. Learners watched Body, Face, or listened to Audio, and worked on six comprehension questions twice for each task. Moreover, the participants were asked about their modality of preference. The notable results were (1) the modality of listening had no influence on their listening performance, (2) for the high proficiency group, the score difference between the easy and the hard texts was larger at the second attempt than at the first attempt, (3) the participants generally preferred to watch the whole body, followed by watching the face, and then listening only, (4) the high proficiency group showed more variations of preference than the low proficiency group, and (5) the participants’ modality preference had no influence on their listening performance.

Beyond the Text: Testing Narrative Persuasion Mechanisms with Audio Messages

ABSTRACT The affective news extended model (Knobloch-Westerwick et al., 2020) demonstrated that narratives with a suspense-inducing format, a linear compared to inverted pyramid format, led to attitude change across four political topics. Given the lack of narrative persuasion research which considers the audio modality, the present research seeks to extend this model by using audio clip stimuli to (1) investigate whether the findings of Knobloch-Westerwick et al. (2020) will extend to the audio modality and (2) compare persuasive mechanisms and outcomes between the textual and audio modalities to understand the unique impact of suspense on narrative persuasion. Results demonstrated that (1) suspense did not differ by format, nor did it mediate the relationship between format and attitude change in the audio modality. (2) Comparisons between textual and audio modalities revealed that when exposed to the audio modality, participants were persuaded by transportation, feelings of being mentally and emotionally involved in the narrative. Future research and implications for the ANEM, audio narratives, and persuasion are discussed.

Modeling Two-Stream Correspondence for Visual Sound Separation

Visual sound separation (VSS) aims to obtain each sound component from the mixed audio signals with the guidance of visual information. Existing works mainly capture the global-level audio-visual correspondence and exploit various visual features to enhance the appearance and motion features of visual modality. However, they commonly neglect the intrinsic properties of the audio modality, resulting in less effective audio feature extraction and unbalanced audio-visual correspondence. To tackle this problem, we propose a novel end-to-end framework termed Modeling Two-Stream Correspondence (MTSC) for VSS by explicitly extracting the timbre and content features in audio modality. The proposed MTSC method employs a two-stream architecture to enhance audio-visual correspondence for both the appearance-timbre and motion-content features. Moreover, with the advanced two-stream pipeline, more lightweight appearance and motion features for visual modality are exploited. Extensive experiments conducted on two benchmark musical instrument datasets demonstrate that with the above properties, our MTSC method remarkably outperforms seven state-of-the-art VSS approaches. The implementation code and extensive experimental results of the proposed MTSC method are provided at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/CFM-MSG/MTSC-VSS</uri> .

Taris: An online speech recognition framework with sequence to sequence neural networks for both audio-only and audio-visual speech

It is widely accepted that the visual modality of speech provides complementary information to the speech recognition task, and many models have been introduced in order to make good use of the visual channel. This article develops Taris, a fully differentiable neural network model capable of decoding both audio-only and audio-visual speech in real time. We achieve this by connecting our previously proposed models AV Align and Taris, which are both end-to-end differentiable approaches to audio-visual speech integration and online speech recognition respectively. We evaluate AV Taris under the same conditions as AV Align and Taris on one of the largest publicly available audio-visual speech datasets, LRS2. Our results show that AV Taris is superior to the audio-only variant of Taris, demonstrating the utility of the visual modality to speech recognition within the real time decoding framework defined by Taris. Compared to an equivalent Transformer-based AV Align model that takes advantage of full sentences without meeting the real-time requirement, we report an absolute degradation of approximately 3% with AV Taris. As opposed to the more popular alternative for online speech recognition, namely the RNN Transducer, Taris offers a greatly simplified fully differentiable training pipeline. We speculate that AV Taris has the potential to popularise the adoption of Audio-Visual Speech Recognition (AVSR) technology and overcome the inherent limitations of the audio modality in less optimal listening conditions.11Our code is publicly available at https://github.com/georgesterpu/Taris.

GAR-Net: A Graph Attention Reasoning Network for conversation understanding

Conversation understanding, as a necessary step for many applications, including social media, education, and argumentation mining, has been gaining increasing attention from the research community. Reasoning over long-term dependent contextual information is the key to utterance-level conversation understanding. Aiming to emphasize the importance of contextual reasoning, an end-to-end graph attention reasoning network which takes both word-level and utterance-level context into concern is proposed. To be specific, a word-level encoder with a novel convolution gate is first built to filter out irrelevant contextual information. Based on the representation extracted by word-level encoder, a graph reasoning network is designed to utilize the context among utterance-level, where the entire conversation is treated as a fully connected graph, utterances as nodes, and attention scores between utterances as edges. The proposed model is a general framework for conversation understanding tasks, which can be flexibly applied on most conversation datasets without changing the network architecture. Furthermore, we revise the tensor fusion network by adding a residual connection to explore cross-modal conversation understanding. For uni-modal scene (text modality), experiments show that the proposed method surpasses current state-of-the-art methods on emotion recognition, intent classification, and dialogue act identification tasks. For cross-modal scenes (text modality and audio modality), experiments on IEMOCAP and MELD datasets show that the proposed method can use cross-modal information to improve model performance.22The code will be available at https://github.com/thuiar/GAR-Net.

Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN

Audio-visual emotion recognition is the research of identifying human emotional states by combining the audio modality and the visual modality simultaneously, which plays an important role in intelligent human-machine interactions. With the help of deep learning, previous works have made great progress for audio-visual emotion recognition. However, these deep learning methods often require a large amount of data for training. In reality, data acquisition is difficult and expensive, especially for the multimodal data with different modalities. As a result, the training data may be in the low-data regime, which cannot be effectively used for deep learning. In addition, class imbalance may occur in the emotional data, which can further degrade the performance of audio-visual emotion recognition. To address these problems, we propose an efficient data augmentation framework by designing a multimodal conditional generative adversarial network (GAN) for audio-visual emotion recognition. Specifically, we design generators and discriminators for audio and visual modalities. The category information is used as their shared input to make sure our GAN can generate fake data of different categories. In addition, the high dependence between the audio modality and the visual modality in the generated multimodal data is modeled based on Hirschfeld-Gebelein-Rényi (HGR) maximal correlation. In this way, we relate different modalities in the generated data to approximate the real data. Then, the generated data are used to augment our data manifold. We further apply our approach to deal with the problem of class imbalance. To the best of our knowledge, this is the first work to propose a data augmentation strategy with a multimodal conditional GAN for audio-visual emotion recognition. We conduct a series of experiments on three public multimodal datasets, including eNTERFACE’05, RAVDESS, and CMEW. The results indicate that our multimodal conditional GAN has high effectiveness for data augmentation of audio-visual emotion recognition.

Speech Driven Talking Face Generation From a Single Image and an Emotion Condition

Visual emotion expression plays an important role in audiovisual speech communication. In this work, we propose a novel approach to rendering visual emotion expression in speech-driven talking face generation. Specifically, we design an end-to-end talking face generation system that takes a speech utterance, a single face image, and a categorical emotion label as input to render a talking face video synchronized with the speech and expressing the conditioned emotion. Objective evaluation on image quality, audiovisual synchronization, and visual emotion expression shows that the proposed system outperforms a state-of-the-art baseline system. Subjective evaluation of visual emotion expression and video realness also demonstrates the superiority of the proposed system. Furthermore, we conduct a human emotion recognition pilot study using generated videos with mismatched emotions among the audio and visual modalities. Results show that humans respond to the visual modality more significantly than the audio modality on this task.

Contextual and Cross-Modal Interaction for Multi-Modal Speech Emotion Recognition

Speech emotion recognition combining linguistic content and audio signals in the dialog is a challenging task. Nevertheless, previous approaches have failed to explore emotion cues in contextual interactions and ignored the long-range dependencies between elements from different modalities. To tackle the above issues, this letter proposes a multimodal speech emotion recognition method using audio and text data. We first present a contextual transformer module to introduce contextual information via embedding the previous utterances between interlocutors, which enhances the emotion representation of the current utterance. Then, the proposed cross-modal transformer module focuses on the interactions between text and audio modalities, adaptively promoting the fusion from one modality to another. Furthermore, we construct associative topological relation over mini-batch and learn the association between deep fused features with graph convolutional network. Experimental results on the IEMOCAP and MELD datasets show that our method outperforms current state-of-the-art methods.

Multimodal Fusion Remote Sensing Image–Audio Retrieval

Remote sensing image-audio retrieval (RSIAR) has been an emerging research topic in recent years, and many different methods have been proposed for this topic. These RSIAR methods have achieved good retrieval results, but two problems remain: the lack of discriminability of audio modality and the existence of a heterogeneous gap between audio and image. These two problems make the cross-modal common embedding space for audio and images suboptimal, often failing to perform superior retrieval. This paper proposes a novel RSIAR method named multi-modal fusion remote sensing image-audio retrieval (MMFR) to address these two problems. MMFR first converts original audio input to text. Then MMFR uses a feature fusion module to obtain a fusion representation fused with text information instead of the original sole audio representation. Fusion text information can make the pronunciation-based audio feature more semantically discriminable and convert pronunciation-based audio feature to more &#x201C;high level&#x201D; fusion feature to cross the heterogeneous gap. Seven different fusion methods are tried in the feature fusion module. In addition, the triplet loss, the semantic loss and the consistency loss are used to optimize the common retrieval space. Extensive experiments conducted on the UCM_IV, RSICD_IV, and SYDNE_IV datasets demonstrate that our MMFR method outperforms state-of-the-art methods.

Multimodal Affect Models: An Investigation of Relative Salience of Audio and Visual Cues for Emotion Prediction

People perceive emotions via multiple cues, predominantly speech and visual cues, and a number of emotion recognition systems utilize both audio and visual cues. Moreover, the perception of static aspects of emotion (speaker's arousal level is high/low) and the dynamic aspects of emotion (speaker is becoming more aroused) might be perceived via different expressive cues and these two aspects are integrated to provide a unified sense of emotion state. However, existing multimodal systems only focus on single aspect of emotion perception and the contributions of different modalities toward modeling static and dynamic emotion aspects are not well explored. In this paper, we investigate the relative salience of audio and video modalities to emotion state prediction and emotion change prediction using a Multimodal Markovian affect model. Experiments conducted in the RECOLA database showed that audio modality is better at modeling the emotion state of arousal and video for emotion state of valence, whereas audio shows superior advantages over video in modeling emotion changes for both arousal and valence.

A multi-modal personality prediction system

The behavior, mental-health, emotion, life choices, social nature, and thought patterns of an individual are revealed by personality. Cyber forensics, personalized services, recommender systems are some of the examples of automatic personality prediction. A deep learning based personality prediction system has been developed in this work. Facial and ambient features are extracted from the visual modality using Multi-task Cascaded Convolutional Networks (MTCNN) and ResNet, respectively; the audio features are extracted using the VGGish Convolutional Neural Networks (VGGish CNN), and the text features are extracted using n-gram Convolutional Neural Networks (CNN). The extracted features are then passed to a fully connected layer followed by sigmoid for the final output prediction. Finally, the text, visual and audio modalities are combined in different ways: (i) concatenation of features in multi-modal setting, and (ii) application of different attention mechanisms for fusing features. The dataset released in Chalearn-17 is used for evaluating the performance of the system. From the obtained results, it can be concluded that, the concatenation of features extracted from different modalities attains comparable results with the averaging method (late fusion). It is also shown that a hand full of images are enough for attaining comparable performance.

Multimodal sentiment analysis with unidirectional modality translation

Multimodal Sentiment Analysis (MSA) is a challenging research area that investigates sentiment expressed from multiple heterogeneous sources of information. To integrate multimodal information including text, visual and audio modalities, state-of-the-art models focus on developing various fusion strategies, such as attention and outer product. However, the inferior quality of visual and audio features that is commonly observed in this area has not aroused much attention. We argue that this issue will obstruct the performance of the fusion strategies to a considerable extent. Therefore, in this paper, we propose Multimodal Translation for Sentiment Analysis (MTSA), a multimodal framework that improves the quality of visual and audio features by translating them to text features extracted by Bidirectional Encoder Representations from Transformers (BERT). Experiments on two benchmark datasets CMU-MOSI and CMU-MOSEI show that our model performs better than the state-of-the-art methods on both datasets across all the metrics, which illustrates the effectiveness of our method.