Audio Input Research Articles

Review of Talking Head Synthesis for Driving Mechanisms and Portrait Rendering

Abstract. Talking head synthesis has emerged as a vital area of research, enabling the generation of realistic and expressive digital avatars. This paper explores the primary mechanisms driving talking head synthesis, categorized into video-driven and audio-driven methods. Video-driven techniques manipulate facial movements using key points, 3D meshes, and latent spaces, while audio-driven approaches focus on synchronizing lip movements and facial expressions with audio inputs. Recent advances in each method, highlighting key innovations and the challenges faced, such as occlusion, identity preservation, and lip synchronization are reviewed. The technology's applications span smart customer service, online education, telemedicine, and video creation. Future research directions focus on overcoming challenges like handling large-angle poses, ensuring temporal consistency, and improving multilingual performance.

Advancements in Natural Language Processing for Human-Computer Interaction

• This systematic review explores the advancements in Natural Language Processing (NLP) for Human-Computer Interaction (HCI) over the period from 2010 to 2024. It highlights the significant breakthroughs achieved through deep learning models, particularly transformer architectures such as BERT and GPT, which have transformed the ability of machines to understand and generate human language. The integration of multimodal capabilities has further enriched user interactions by enabling the processing of diverse data types, including text, audio, and visual inputs. However, the review also identifies persistent challenges, including maintaining coherence in long dialogues, resolving ambiguous language, addressing bias in training data, and the need for resource-efficient models. Additionally, the paper emphasizes the importance of cross-lingual capabilities for low-resource languages and the necessity of personalized, adaptive systems. The findings underscore the need for ongoing research to overcome existing limitations and enhance the effectiveness and inclusivity of NLP technologies in HCI, ultimately contributing to a more intuitive and accessible user experience.

RADIANCE: Reliable and interpretable depression detection from speech using transformer

Depression is a common but severe mental disorder that adversely impacts the ability of an individual to function normally in their day-to-day life. A majority of depressed individuals remain undiagnosed due to factors such as social stigma and a shortage of healthcare professionals. Consequently, several Machine Learning and Deep Learning (DL) models based on speech have been proposed for automatic depression detection, with the latter generally outperforming the former. However, DL models are blackbox and offer no transparency. In contrast, healthcare professionals prefer models that provide interpretability besides being accurate. In this direction, we propose a method RADIANCE (Reliable AnD InterpretAble depressioN deteCtion transformErs). RADIANCE incorporates a novel FilterBank VIsion Transformer (FBViT) network, which provides the symptoms of depression as interpretable features. Additionally, we employ a novel loss function that handles the class imbalance issue in the datasets. It also incorporates a penalty term that addresses the hierarchy of misclassification errors. We also propose a reliability predictor based on low-level descriptors that provides a reliability score to indicate the trustworthiness of the prediction by FBViT. Furthermore, in contrast to the conventional averaging and majority pooling, RADIANCE consolidates predictions from multiple clips of the input audio by intricately weighing each prediction based on its reliability score, ensuring a more accurate overall prediction. RADIANCE outperforms the state-of-the-art depression detection methods, achieving an accuracy of 89.36%, 80.36%, and 94.44% over the DAIC-WOZ, E-DAIC, and CMDC datasets, respectively. Further, RADIANCE achieves MAE scores of 3.27 and 5.04 on the DAIC-WOZ and E-DAIC datasets, respectively.

Hierarchical-Concatenate Fusion TDNN for sound event classification.

Semantic feature combination/parsing issue is one of the key problems in sound event classification for acoustic scene analysis, environmental sound monitoring, and urban soundscape analysis. The input audio signal in the acoustic scene classification is composed of multiple acoustic events, which usually leads to low recognition rate in complex environments. To address this issue, this paper proposes the Hierarchical-Concatenate Fusion(HCF)-TDNN model by adding HCF Module to ECAPA-TDNN model for sound event classification. In the HCF module, firstly, the audio signal is converted into two-dimensional time-frequency features for segmentation. Then, the segmented features are convolved one by one for improving the small receptive field in perceiving details. Finally, after the convolution is completed, the two adjacent parts are combined before proceeding with the next convolution for enlarging the receptive field in capturing large targets. Therefore, the improved model further enhances the scalability by emphasizing channel attention and efficient propagation and aggregation of feature information. The proposed model is trained and validated on the Urbansound8K dataset. The experimental results show that the proposed model can achieve the best classification accuracy of 95.83%, which is an approximate improvement of 5% (relatively) over the ECAPA-TDNN model.

Audio-Driven Facial Animation with Deep Learning: A Survey

Audio-driven facial animation is a rapidly evolving field that aims to generate realistic facial expressions and lip movements synchronized with a given audio input. This survey provides a comprehensive review of deep learning techniques applied to audio-driven facial animation, with a focus on both audio-driven facial image animation and audio-driven facial mesh animation. These approaches employ deep learning to map audio inputs directly onto 3D facial meshes or 2D images, enabling the creation of highly realistic and synchronized animations. This survey also explores evaluation metrics, available datasets, and the challenges that remain, such as disentangling lip synchronization and emotions, generalization across speakers, and dataset limitations. Lastly, we discuss future directions, including multi-modal integration, personalized models, and facial attribute modification in animations, all of which are critical for the continued development and application of this technology.

Audio Deep Fake Detection with Sonic Sleuth Model

Information dissemination and preservation are crucial for societal progress, especially in the technological age. While technology fosters knowledge sharing, it also risks spreading misinformation. Audio deepfakes—convincingly fabricated audio created using artificial intelligence (AI)—exacerbate this issue. We present Sonic Sleuth, a novel AI model designed specifically for detecting audio deepfakes. Our approach utilizes advanced deep learning (DL) techniques, including a custom CNN model, to enhance detection accuracy in audio misinformation, with practical applications in journalism and social media. Through meticulous data preprocessing and rigorous experimentation, we achieved a remarkable 98.27% accuracy and a 0.016 equal error rate (EER) on a substantial dataset of real and synthetic audio. Additionally, Sonic Sleuth demonstrated 84.92% accuracy and a 0.085 EER on an external dataset. The novelty of this research lies in its integration of datasets that closely simulate real-world conditions, including noise and linguistic diversity, enabling the model to generalize across a wide array of audio inputs. These results underscore Sonic Sleuth’s potential as a powerful tool for combating misinformation and enhancing integrity in digital communications.

Enhancing human computer interaction with coot optimization and deep learning for multi language identification

Human-Computer Interaction (HCI) is a multidisciplinary field focused on designing and utilizing computer technology, underlining the interaction interface between computers and humans. HCI aims to generate systems that allow consumers to relate to computers effectively, efficiently, and pleasantly. Multiple Spoken Language Identification (SLI) for HCI (MSLI for HCI) denotes the ability of a computer system to recognize and distinguish various spoken languages to enable more complete and handy interactions among consumers and technology. SLI utilizing deep learning (DL) involves using artificial neural networks (ANNs), a subset of DL models, to automatically detect and recognize the language spoken in an audio signal. DL techniques, particularly neural networks (NNs), have succeeded in various pattern detection tasks, including speech and language processing. This paper develops a novel Coot Optimizer Algorithm with a DL-Driven Multiple SLI and Detection (COADL-MSLID) technique for HCI applications. The COADL-MSLID approach aims to detect multiple spoken languages from the input audio regardless of gender, speaking style, and age. In the COADL-MSLID technique, the audio files are transformed into spectrogram images as a primary step. Besides, the COADL-MSLID technique employs the SqueezeNet model to produce feature vectors, and the COA is applied to the hyperparameter range of the SqueezeNet method. The COADL-MSLID technique exploits the SLID process’s convolutional autoencoder (CAE) model. To underline the importance of the COADL-MSLID technique, a series of experiments were conducted on the benchmark dataset. The experimentation validation of the COADL-MSLID technique exhibits a greater accuracy result of 98.33% over other techniques.

Talking Face Generation With Audio-Deduced Emotional Landmarks.

The goal of talking face generation is to synthesize a sequence of face images of the specified identity, ensuring the mouth movements are synchronized with the given audio. Recently, image-based talking face generation has emerged as a popular approach. It could generate talking face images synchronized with the audio merely depending on a facial image of arbitrary identity and an audio clip. Despite the accessible input, it forgoes the exploitation of the audio emotion, inducing the generated faces to suffer from emotion unsynchronization, mouth inaccuracy, and image quality deficiency. In this article, we build a bistage audio emotion-aware talking face generation (AMIGO) framework, to generate high-quality talking face videos with cross-modally synced emotion. Specifically, we propose a sequence-to-sequence (seq2seq) cross-modal emotional landmark generation network to generate vivid landmarks, whose lip and emotion are both synchronized with input audio. Meantime, we utilize a coordinated visual emotion representation to improve the extraction of the audio one. In stage two, a feature-adaptive visual translation network is designed to translate the synthesized landmarks into facial images. Concretely, we proposed a feature-adaptive transformation module to fuse the high-level representations of landmarks and images, resulting in significant improvement in image quality. We perform extensive experiments on the multi-view emotional audio-visual dataset (MEAD) and crowd-sourced emotional multimodal actors dataset (CREMA-D) benchmark datasets, demonstrating that our model outperforms state-of-the-art benchmarks.

Achieving liquid processors by colloidal suspensions for reservoir computing

The increasing use of machine learning, with its significant computational and environmental costs, has motivated the exploration of unconventional computing substrates. Liquid substrates, such as colloids, are of particular interest due to their ability to conform to various shapes while exhibiting complex dynamics resulting from the collective behaviour of the constituent colloidal particles. This study explores the potential of using a PEDOT:PSS colloidal suspension as a physical reservoir for reservoir computing in spoken digit recognition. Reservoir computing uses high-dimensional dynamical systems to perform tasks with different substrates, including physical ones. Here, a physical reservoir is implemented that encodes temporal data by exploiting the rich dynamics inherent in colloidal suspensions, thus avoiding reliance on conventional computing hardware. The reservoir processes audio input encoded as spike sequences, which are then classified using a trained readout layer to identify spoken digits. Evaluation across different speaker scenarios shows that the colloidal reservoir achieves high accuracy in classification tasks, demonstrating its viability as a physical reservoir substrate.

3D Visual Grounding-Audio: 3D scene object detection based on audio

3D Visual Grounding (3DVG) is a prevalent multi-modal information fusion task capable of accurately localizing target objects referenced in natural language descriptions within a point cloud scene. Nevertheless, the stringent demands for input and output devices present substantial hurdles for the application and integration of 3DVG in fields like remote robotic control and telemedicine. To address this challenge, we introduce several innovative approaches. Firstly, we have initiated a novel multi-modal task, termed 3D Visual Grounding-Audio (3DVG-Audio), which is based on the fusion of audio and point cloud. To the best of our knowledge, this represents the first instance of an Audio-Point Cloud multi-modal task. 3DVG-Audio achieves precise localization of audio-mentioned objects within the point cloud by utilizing the point cloud in conjunction with the corresponding audio input. Secondly, building upon the ScanRefer, we have developed a novel dataset named 3DVG-AudioSet, specifically designed for the training and evaluation of the 3DVG-Audio method. Finally, we have crafted a tailored loss function to further enhance the performance of 3DVG-Audio and introduced a method named AP-Refer, which serves as a benchmark for the task. Extensive experimental results demonstrate the potential for deep integration of audio and point cloud to tackle complex real-world challenges. AP-Refer has successfully addressed the 3DVG-Audio, circumventing the limitations of conventional 3DVG methods, and exhibits significant application potential.

Research on Multi-Modal Music Score Alignment Model for Online Music Education

As music data storage becomes increasingly diverse in the era of big data, ensuring alignment of music works with the same semantics for online music education is crucial. To achieve this, a multi-modal music score alignment algorithm model based on deep learning was developed and optimized. Experimental results demonstrated that Note + DCO feature combination yielded the best MIDI input characteristics (mean value: 13.27 ms), whereas CQT feature comparison produced the best results for audio input (average: 12.85 ms). The ResNet-34 network was noted to have the most effective music score alignment effect with alignment errors averaging less than one frame. Compared with other algorithms, the proposed algorithm had the lowest average value of 9.28 ms, median value of 5.85 ms, and standard deviation of 20.17 ms. Actual music retrieval showed a Top-1 retrieval accuracy of 10.93% that was close to 11%. Overall, the proposed algorithm is significant for score alignment and music retrieval recognition in online music education.

Depression Detection with Novel Large Language Model Methods

Recent advancements in machine learning and natural language processing have shown significant potential in aiding mental health diagnostics. In this paper, we specifically provide novel approaches to depression identification using large language models (LLMs). First, we employ the LLM2Vec architecture for text-only classification. Additionally, we explore the potential of multimodal LLMs by incorporating text, audio, and visual inputs, aiming to enhance diagnostic accuracy through a more comprehensive analysis of multimodal data. Our results demonstrate that text-only models perform well above baseline performances, while multimodal data shows promise for achieving a more nuanced understanding of a patient’s mental state. This work lays the groundwork for future research in developing more robust and effective tools for mental health assessment using LLMs.

Headphones over the cochlear-implant sound processor to replace direct audio input.

Psychoacoustic stimulus presentation to the cochlear implant via direct audio input (DAI) is no longer possible for many newer sound processors (SPs). This study assessed the feasibility of placing circumaural headphones over the SP. Calibration spectra for loudspeaker, DAI, and headphone modalities were estimated by measuring cochlear-implant electrical output levels for tones presented to SPs on an acoustic manikin. Differences in calibration spectra between modalities arose mainly from microphone-response characteristics (high-frequency differences between DAI and the other modalities) or a proximity effect (low-frequency differences between headphones and loudspeaker). Calibration tables are provided to adjust for differences between the three modalities.

Speed-Aware Audio-Driven Speech Animation using Adaptive Windows

We present a novel method that can generate realistic speech animations of a 3D face from audio using multiple adaptive windows. In contrast to previous studies that use a fixed size audio window, our method accepts an adaptive audio window as input, reflecting the audio speaking rate to use consistent phonemic information. Our system consists of three parts. First, the speaking rate is estimated from the input audio using a neural network trained in a self-supervised manner. Second, the appropriate window size that encloses the audio features is predicted adaptively based on the estimated speaking rate. Another key element lies in the use of multiple audio windows of different sizes as input to the animation generator: a small window to concentrate on detailed information and a large window to consider broad phonemic information near the center frame. Finally, the speech animation is generated from the multiple adaptive audio windows. Our method can generate realistic speech animations from in-the-wild audios at any speaking rate, i.e., fast raps, slow songs, as well as normal speech. We demonstrate via extensive quantitative and qualitative evaluations including a user study that our method outperforms state-of-the-art approaches.

Audio-guided implicit neural representation for local image stylization

We present a novel framework for audio-guided localized image stylization. Sound often provides information about the specific context of a scene and is closely related to a certain part of the scene or object. However, existing image stylization works have focused on stylizing the entire image using an image or text input. Stylizing a particular part of the image based on audio input is natural but challenging. This work proposes a framework in which a user provides an audio input to localize the target in the input image and another to locally stylize the target object or scene. We first produce a fine localization map using an audio-visual localization network leveraging CLIP embedding space. We then utilize an implicit neural representation (INR) along with the predicted localization map to stylize the target based on sound information. The INR manipulates local pixel values to be semantically consistent with the provided audio input. Our experiments show that the proposed framework outperforms other audio-guided stylization methods. Moreover, we observe that our method constructs concise localization maps and naturally manipulates the target object or scene in accordance with the given audio input.

Kombucha–Chlorella–Proteinoid Biosynthetic Classifiers of Audio Signals

ABSTRACTThis paper describes the development of a bioinspired composite material capable of audio classification applications. Hydrogel matrices produced by microorganisms combined with synthetic biology elements, allow for the development of adaptable bioelectronics that connect biology and technology in a customized way. In this study, a composite population of kombucha, chlorella, and proteinoids (thermal proteins) is utilized to respond to acoustic signals converted to electrical waveforms. The kombucha zoogleal mats, which are made and populated by over 60 species of yeasts and bacteria, offer a matrix at the micro level that is connected to the photosynthetic microalgae chlorella. Proteinoids formed through thermal condensation exhibit unique patterns of signaling kinetics. This living material has the ability to be electrically stimulated and can process signals in a way feasible for sensory applications. Using English alphabet audio inputs, a systematic analysis demonstrates the capability to differentiate audio waveforms based solely on biological composite responses. The use of spectral analysis allows for the identification of specific spike timing patterns that encode unique characteristics of individual letters. Moreover, network disturbances result in specific changes in output, so validating the ability to adjust waveform classification. The study demonstrates that kombucha–chlorella–proteinoid composites provide a durable and versatile bioelectronic platform for immediate auditory processing. The work represents progress toward the development of bioelectronic systems that can be customized based on the principles of biological sensory processing, cognition, and adaptation.

Semantic-aware hyper-space deformable neural radiance fields for facial avatar reconstruction

High-fidelity facial avatar reconstruction from monocular videos is a prominent research problem in computer graphics and computer vision. Recent advancements in the Neural Radiance Field (NeRF) have demonstrated remarkable proficiency in rendering novel views and garnered attention for its potential in facial avatar reconstruction. However, previous methodologies have overlooked the complex motion dynamics present across the head, torso, and intricate facial features. Additionally, a deficiency exists in a generalized NeRF-based framework for facial avatar reconstruction adaptable to either 3DMM coefficients or audio input. To tackle these challenges, we propose an innovative framework that leverages semantic-aware hyper-space deformable NeRF, facilitating the reconstruction of high-fidelity facial avatars from either 3DMM coefficients or audio features. Our framework effectively addresses both localized facial movements and broader head and torso motions through semantic guidance and a unified hyper-space deformation module. Specifically, we adopt a dynamic weighted ray sampling strategy to allocate varying degrees of attention to distinct semantic regions, enhancing the deformable NeRF framework with semantic guidance to capture fine-grained details across diverse facial regions. Moreover, we introduce a hyper-space deformation module that enables the transformation of observation space coordinates into canonical hyper-space coordinates, allowing for the learning of natural facial deformation and head-torso movements. Extensive experiments validate the superiority of our framework over existing state-of-the-art methods, demonstrating its effectiveness in producing realistic and expressive facial avatars. Our code is available at https://github.com/jematy/SAHS-Deformable-Nerf.

KUMPAS: A Filipino Sign Language Translator of Deaf and Hearing Individuals Utilizing Computer Vision Algorithm

This study elucidates how the Filipino “Kumpas” translator improves communication between signers and non-signers. Many Hearing people are likely familiar with American Sign Language (ASL), but many are still unaware that the Filipino Deaf have their own unique language called Filipino Sign Language (FSL). In this section, the programmers present a method for bridging the gap between signers and non-signers. Sign to text and speech to sign are the two modes of communication for the “Kumpas” translator. In order to translate using speech, the application would use a speech audio input and based from the words recognized, a 3D character will play a sequence of animations according to the translation. The OpenCV Python library, which is useful for computer vision and image processing tasks, was used by the programmers. MediaPipe has a number of features, including tracking, face recognition, and object detection. The system was tested using an ISO 25010 and based on the evaluation, an overall very satisfactory rating was obtained. Also, findings presented that the system is definitely a useful language translator for Filipinos.

Adversarial alignment and graph fusion via information bottleneck for multimodal emotion recognition in conversations

With the rapid development of social media and human–computer interaction, multimodal emotion recognition in conversations (MERC) tasks have begun to receive widespread research attention. The MERC task is to extract and fuse complementary semantic information from different modalities to classify the speaker’s emotion. However, the existing feature fusion methods usually directly map the features of other modalities into the same feature space for information fusion, which cannot eliminate the heterogeneity between different modalities and make the subsequent emotion class boundary learning more difficult. In addition, existing graph contrastive learning methods obtain consistent feature representations by maximizing mutual information between multiple views, which may lead to overfitting of the model. To tackle the above problem, we propose a novel Adversarial Alignment and Graph Fusion via Information Bottleneck for Multimodal Emotion Recognition in Conversations (AGF-IB) method. Firstly, we input video, audio, and text features into a multi-layer perceptron (MLP) to map them into separate feature spaces. Secondly, we build a generator and a discriminator for the three modal features, respectively, through adversarial representation to achieve information interaction between modalities and eliminate the heterogeneity among modalities. Thirdly, we introduce graph contrastive representation learning to capture intra-modal and inter-modal complementary semantic information and learn intra-class and inter-class boundary information of emotion categories. Furthermore, instead of maximizing the mutual information (MI) between multiple views, we use information bottleneck theory to minimize the MI between views. Specifically, we construct a graph structure for the three modal features respectively and perform contrastive representation learning on nodes with different emotions in the same modality and nodes with the same emotion in different modalities, to improve the feature representation ability of nodes. Finally, we use MLP to complete the emotional classification of the speaker. Extensive experiments show that AGF-IB can improve emotion recognition accuracy on IEMOCAP and MELD datasets. Furthermore, since AGF-IB is a general multimodal fusion and contrastive learning method, it can be applied to other multimodal tasks in a plug-and-play manner, e.g., humor detection.

Duration of cochlear implant use in children with prelingual single-sided deafness is a predictor of word perception in the CI ear

As part of a longitudinal study regarding the benefit of early cochlear implantation for children with single-sided deafness, the current work explored the children's daily device use, potential barriers to full-time device use, and the children's ability to understand speech with the cochlear implant (CI). Data were collected from 20 children with prelingual SSD who received a CI before the age of 2.5 years, from the initial activation of the sound processor until the children were 4.8 to 11.0 years old. Daily device use was extracted from the CI's data logging, while word perception in quiet was assessed using direct audio input to the children's sound processor. The children's caregivers completed a questionnaire about habits, motivations, and barriers to device use.The children with SSD and a CI used their device on average 8.3 h per day, corresponding to 63 % of their time spent awake. All children except one could understand speech through the CI, with an average score of 59 % on a closed-set test and 73 % on an open-set test. More device use was associated with higher speech perception scores. Parents were happy with their decision to pursue a CI for their child. Certain habits, like taking off the sound processor during illness, were associated with lower device use. Providing timely counselling to the children's parents, focused on SSD-specific challenges, may be helpful to improve daily device use in these children.