Audio Context Research Articles

Audio-to-Deep-Lip: Speaking lip synthesis based on 3D landmarks

Generating talking lips in sync with input speech has the potential to enhance speech communication and enable novel applications. This paper presents a system that can generate accurate 3D talking lips, readily applicable to unseen subjects and different languages. The developed head-mounted facial acquisition device and automated data processing pipeline can generate precise landmarks while mitigating the difficulty of acquiring 3D facial data. Our system consists of three stages to generate accurate lip movements. In the first stage, the fine-tuned Wav2Vec2.0+Transformer captures long-range audio context dependencies. In the second stage, we propose the Viseme Fixing method, which significantly improves lip accuracy at/b//p//m//f/ phonemes. In the last stage, we innovatively use the structural relationship between the inner and outer lips and learn to map the outer lip landmarks to the inner lip landmarks. Subjective evaluations show that the generated talking lips match the input audio significantly. We demonstrate two applications that animate 2D face videos and 3D face models using our landmarks. The precise lip landmarks allow the generated animations to exceed the results of state-of-the-art methods.

Slow looking at still art: The effect of manipulating audio context and image category on mood and engagement during an online slow looking exercise.

Slow looking at still art: The effect of manipulating audio context and image category on mood and engagement during an online slow looking exercise.

Frequency-Invariant Sensor Selection for MVDR Beamforming in Wireless Acoustic Sensor Networks

Wireless acoustic sensor network (WASN) has a wide range of applications in internet of things, where signal estimation is one of the network design objectives. Due to the existence of ambient noises, the recorded audio signals are inevitably corrupted, resulting in a low signal-to-noise ratio (SNR), which triggers the necessity of signal enhancement. As using all sensor measurements brings a large amount of data transmissions and computational cost, the narrowband sensor selection was proposed to choose an informative subset of sensors to perform noise reduction in the audio context. However, the resulting frequency-dependent selection status has to be switched across frequencies. In order to avoid the complicated switching operations, we consider frequency-invariant sensor selection in this work. We propose to minimize the total power consumption over the WASN by constraining the broadband SNR, which can be solved using broadband semi-definite optimization (BroadOpt) or narrowband voting (NaVo) approaches. In order to further reduce the time complexity, we propose two near-optimal greedy methods, including gradient removal (GradR) and weighted input SNR removal (SnrR). As comparison, we also show a broadband energy removal (EnergyR) method. The greedy methods remove one sensor at each iteration from the complete network until the performance constraint is not satisfied. Numerical results using a simulated large-scale WASN show that the greedy methods can achieve a comparable performance compared to the optimization based counterparts, while the corresponding time complexity is much lower. In general, the sensors around the target source and the fusion center are more likely to be selected.

Music with Concurrent Saliences of Musical Features Elicits Stronger Brain Responses

Brain responses are often studied under strictly experimental conditions in which electroencephalograms (EEGs) are recorded to reflect reactions to short and repetitive stimuli. However, in real life, aural stimuli are continuously mixed and cannot be found isolated, such as when listening to music. In this audio context, the acoustic features in music related to brightness, loudness, noise, and spectral flux, among others, change continuously; thus, significant values of these features can occur nearly simultaneously. Such situations are expected to give rise to increased brain reaction with respect to a case in which they would appear in isolation. In order to assert this, EEG signals recorded while listening to a tango piece were considered. The focus was on the amplitude and time of the negative deflation (N100) and positive deflation (P200) after the stimuli, which was defined on the basis of the selected music feature saliences, in order to perform a statistical analysis intended to test the initial hypothesis. Differences in brain reactions can be identified depending on the concurrence (or not) of such significant values of different features, proving that coterminous increments in several qualities of music influence and modulate the strength of brain responses.

Live 4 Life: A spatial performance tool to play the ephemeral and improvise with space and playback speeds

This article describes my own way to improvise with space using a computer-based tool implemented in SuperCollider. The objective of this spatial performance tool is to have an ergonomic spatio-temporal and spectral control over numerous sound objects in real time, in order to alternate between spatialised polyrhythms and textures. After a brief review of spatial audio context, the spatial performance tool is summarised and detailed here by focusing on one of the core parameters: the playback speeds, which can act both on rhythm and space and enable among others the spatio-temporal articulation of the performance. As well as discussing the word ‘comprovisation’ and my conception of human–computer improvisation, the possibilities and approach of the tool in terms of improvisation and controllerism are illustrated through the use and combination of different controllers (computer keyboard, tactile interfaces, force touch sensors). Whereas some controllers are more dedicated to the selection and triggering of streams of spatialised sound events, others have their own mappings and ways of acting on some parameters (depending on the temporality of the sounds: playing or future events).

An automatic approach of audio feature engineering for the extraction, analysis and selection of descriptors

Currently, it is critical to find the correct features from the audio, in order to analyze the information contained in it. This paper analyzes several feature types in audio from different points of view: time series, sound engineering, etc. In particular, the description of audio as a set of time series is not very common in the literature, and it is one of the aspects studied in this paper. Particularly, this paper proposes an automated method for feature engineering in audios, to extract, analyze and select the best features in a given context. Specifically, this paper develops a hybrid scheme of extraction of audio descriptors based on different principles and defines an automatic approach for the analysis and selection of these descriptors in a given audio context. Finally, our approach was tested on grouping tasks and compared to previous works on audio classification problems, with encouraging results.

Speech gesture generation from the trimodal context of text, audio, and speaker identity

For human-like agents, including virtual avatars and social robots, making proper gestures while speaking is crucial in human-agent interaction. Co-speech gestures enhance interaction experiences and make the agents look alive. However, it is difficult to generate human-like gestures due to the lack of understanding of how people gesture. Data-driven approaches attempt to learn gesticulation skills from human demonstrations, but the ambiguous and individual nature of gestures hinders learning. In this paper, we present an automatic gesture generation model that uses the multimodal context of speech text, audio, and speaker identity to reliably generate gestures. By incorporating a multimodal context and an adversarial training scheme, the proposed model outputs gestures that are human-like and that match with speech content and rhythm. We also introduce a new quantitative evaluation metric for gesture generation models. Experiments with the introduced metric and subjective human evaluation showed that the proposed gesture generation model is better than existing end-to-end generation models. We further confirm that our model is able to work with synthesized audio in a scenario where contexts are constrained, and show that different gesture styles can be generated for the same speech by specifying different speaker identities in the style embedding space that is learned from videos of various speakers. All the code and data is available at https://github.com/ai4r/Gesture-Generation-from-Trimodal-Context.

Audio–Visual Speech Recognition Based on Dual Cross-Modality Attentions with the Transformer Model

Since attention mechanism was introduced in neural machine translation, attention has been combined with the long short-term memory (LSTM) or replaced the LSTM in a transformer model to overcome the sequence-to-sequence (seq2seq) problems with the LSTM. In contrast to the neural machine translation, audio–visual speech recognition (AVSR) may provide improved performance by learning the correlation between audio and visual modalities. As a result that the audio has richer information than the video related to lips, AVSR is hard to train attentions with balanced modalities. In order to increase the role of visual modality to a level of audio modality by fully exploiting input information in learning attentions, we propose a dual cross-modality (DCM) attention scheme that utilizes both an audio context vector using video query and a video context vector using audio query. Furthermore, we introduce a connectionist-temporal-classification (CTC) loss in combination with our attention-based model to force monotonic alignments required in AVSR. Recognition experiments on LRS2-BBC and LRS3-TED datasets showed that the proposed model with the DCM attention scheme and the hybrid CTC/attention architecture achieved at least a relative improvement of 7.3% on average in the word error rate (WER) compared to competing methods based on the transformer model.

Acoustic-decoy: Detection of adversarial examples through audio modification on speech recognition system

Deep neural networks (DNNs) display good performance in the domains of recognition and prediction, such as on tasks of image recognition, speech recognition, video recognition, and pattern analysis. However, adversarial examples, created by inserting a small amount of noise into the original samples, can be a serious threat because they can cause misclassification by the DNN. Adversarial examples have been studied primarily in the context of images, but their effect in the audio context is now drawing considerable interest as well. For example, by adding a small distortion to an original audio sample, imperceptible to humans, an audio adversarial example can be created that humans hear as error-free but that causes misunderstanding by a machine. Therefore, it is necessary to create a method of defense for resisting audio adversarial examples. In this paper, we propose an acoustic-decoy method for detecting audio adversarial examples. Its key feature is that it adds well-formalized distortions using audio modification that are sufficient to change the classification result of an adversarial example but do not affect the classification result of an original sample. Experimental results show that the proposed scheme can detect adversarial examples by reducing the similarity rate for an adversarial example to 6.21%, 1.27%, and 0.66% using low-pass filtering (with 12 dB roll-off), 8-bit reduction, and audio silence removal techniques, respectively. It can detect an audio adversarial example with a success rate of 97% by performing a comparison with the initial audio sample.

Dialogue systems with audio context

Research on building dialogue systems that converse with humans naturally has recently attracted a lot of attention. Most work on this area assumes text-based conversation, where the user message is modeled as a sequence of words in a vocabulary. Real-world human conversation, in contrast, involves other modalities, such as voice, facial expression and body language, which can influence the conversation significantly in certain scenarios. In this work, we explore the impact of incorporating the audio features of the user message into generative dialogue systems. Specifically, we first design an auxiliary response retrieval task for audio representation learning. Then, we use word-level modality fusion to incorporate the audio features as additional context to our main generative model. Experiments show that our audio-augmented model outperforms the audio-free counterpart on perplexity, response diversity and human evaluation.

오디오 컨텍스트의 선호 경향 및 활동 패턴 분석 기법

최근엔 SNS, UCC, 디지털 다이어리 등의 개인 컨텍스트가 많아지고 있으며, 그 종류도 다양화되고 있다. 이에 따라 개인 컨텍스트를 통한 시맨틱 분석에 대한 관심이 증가하고 있다. 시맨틱 분석을 통해 개인 컨텍스트로부터 라이프 스타일, 삶의 질 등의 개인이 인지하지 못하는 다양한 측면을 추론해 낼 수 있다. 본 논문에서는, 개인 컨텍스트 중 오디오 컨텍스트에 대한 시맨틱 분석의 방법론을 제안한다. 특히, 시맨틱 분석의 종류 중 하나인 개인 선호도와 행동 패턴 분석에 대하여 상세한 알고리즘을 제안한다. 그리고, 제안된 기법을 평가하기 위하여, 스마트 다이어리 시스템 개발에 시맨틱 분석 프로세스를 적용하고, 실험한 평가 결과를 제시한다. 제안된 기법을 평가한다. 제안하는 알고리즘을 통한 개인 컨텍스트의 분석 결과는 개인 비서 서비스, 추천 서비스, 광고 서비스 등의 다양한 분야에 활용될 수 있을 것으로 기대된다.

A Study on Psychological Care for the Elderly Using Web-Based Embodied Conversational Agent

In the whole world, the proportion of aged population is increasing day by day. To fulfill Confucius' philosophy of ＂treat others' elderly and youth as if they are our family members,＂ we proposed an intelligent web-based agent which is able to conduct daily life conversation with the elders. Based on psychological theory, we classify the participants of the experiment into three types using K-means. A decision tree is trained based on the information gain during the process of classification. We also implemented a web-based software agent which is visually alive and is able to interact with the elders through visual and audio context. Such agent can be accessed virtually from any location on the planet and is able to have conversation and interaction with the elders to keep them from being bored. The trained decision tree serves as the brain of the web-based agent to classify the type of the elders as the conversation goes on and provide the appropriate questions or choices for the human-machine interaction. Through large-scale experiments we found that the combination of web-based agent and the trained decision tree works satisfactorily for the purpose of caregiving to the elderly.

Context-based environmental audio event recognition for scene understanding

Automatic audio content recognition has attracted an increasing attention for developing multimedia systems, for which the most popular approaches combine frame-based features with statistic models or discriminative classifiers. The existing methods are effective for clean single-source event detection but may not perform well for unstructured environmental sounds, which have a broad noise-like flat spectrum and a diverse variety of compositions. We present an automatic acoustic scene understanding framework that detects audio events through two hierarchies, acoustic scene recognition and audio event recognition, in which the former is preceded by following dominant audio sources and in turn helps infer non-dominant audio events within the same scene through modeling their occurrence correlations. On the scene recognition hierarchy, we perform adaptive segmentation and feature extraction for every input acoustic scene stream through Eigen-audiospace and an optimized feature subspace, respectively. After filtering background, scene streams are recognized by modeling the observation density of dominant features using a two-level hidden Markov model. On the audio event recognition hierarchy, scene knowledge is characterized by an audio context model that essentially describes the occurrence correlations of dominant and non-dominant audio events within this scene. Monte Carlo integration and gradient descent techniques are employed to maximize the likelihood and correctly tag each audio event. To the best of our knowledge, this is the first work that models event correlations as scene context for robust audio event detection from complex and noisy environments. Note that according to the recent report, the mean accuracy for the acoustic scene classification task by human listeners is only around 71 % on the data collected in office environments from the DCASE dataset. None of the existing methods performs well on all scene categories and the average accuracy of the best performances of the recent 11 methods is 53.8 %. The proposed method averagely achieves an accuracy of 62.3 % on the same dataset. Additionally, we create a 10-CASE dataset by manually collecting 5,250 audio clips of 10 scene types and 21 event categories. Our experimental results on 10-CASE show that the proposed method averagely achieves the enhanced performance of 78.3 %, and the average accuracy of audio event recognition can be effectively improved by capturing dominant audio sources and reasoning non-dominant events from the dominant ones through acoustic context modeling. In the future work, exploring the interactions between acoustic scene recognition and audio event detection, and incorporating other modalities to improve the accuracy are required to further advance the proposed framework.

Realistic Human Action Recognition With Multimodal Feature Selection and Fusion

Although promising results have been achieved for human action recognition under well-controlled conditions, it is very challenging to recognize human actions in realistic scenarios due to increased difficulties such as dynamic backgrounds. In this paper, we propose to take multimodal (i.e., audiovisual) characteristics of realistic human action videos into account in human action recognition for the first time, since, in realistic scenarios, audio signals accompanying an action generally provide a cue to the nature of the action, such as phone ringing to answering the phone . In order to cope with diverse audio cues of an action in realistic scenarios, we propose to identify effective features from a large number of audio features with the generalized multiple kernel learning algorithm. The widely used space-time interest point descriptors are utilized as visual features, and a support vector machine is employed for both audio- and video-based classifications. At the final stage, fuzzy integral is utilized to fuse recognition results of both audio and visual modalities. Experimental results on the challenging Hollywood-2 Human Action data set demonstrate that the proposed approach is able to achieve better recognition performance improvement than that of integrating scene context. It is also discovered how audio context influences realistic action recognition from our comprehensive experiments.

Experiences with context management in emergency medicine

In emergency medicine, patient care is intense and stressful, often requiring paramedics to consult with remote physicians to convey the patient's condition. We present a framework for context-management in telemedicine developed in collaboration between engineers, physicians, and paramedics. We describe a mobile platform and embedded wireless sensors to capture physiological and audio context into a comprehensive patient record, accessible locally and remotely. We describe a first evaluation of this technology by trained paramedics in simulated scenarios and evaluate key aspects of system performance. Early results suggest that wireless sensing can provide reliable and low latency data both locally and to remote physicians. In addition, audio context capture is a promising approach to capturing a comprehensive patient record, with a low rate of medically important errors.

Secure Communication Based on Ambient Audio

We propose to establish a secure communication channel among devices based on similar audio patterns. Features from ambient audio are used to generate a shared cryptographic key between devices without exchanging information about the ambient audio itself or the features utilized for the key generation process. We explore a common audio-fingerprinting approach and account for the noise in the derived fingerprints by employing error correcting codes. This fuzzy-cryptography scheme enables the adaptation of a specific value for the tolerated noise among fingerprints based on environmental conditions by altering the parameters of the error correction and the length of the audio samples utilized. In this paper, we experimentally verify the feasibility of the protocol in four different realistic settings and a laboratory experiment. The case studies include an office setting, a scenario where an attacker is capable of reproducing parts of the audio context, a setting near a traffic loaded road, and a crowded canteen environment. We apply statistical tests to show that the entropy of fingerprints based on ambient audio is high. The proposed scheme constitutes a totally unobtrusive but cryptographically strong security mechanism based on contextual information.

Context-dependent sound event detection

The work presented in this article studies how the context information can be used in the automatic sound event detection process, and how the detection system can benefit from such information. Humans are using context information to make more accurate predictions about the sound events and ruling out unlikely events given the context. We propose a similar utilization of context information in the automatic sound event detection process. The proposed approach is composed of two stages: automatic context recognition stage and sound event detection stage. Contexts are modeled using Gaussian mixture models and sound events are modeled using three-state left-to-right hidden Markov models. In the first stage, audio context of the tested signal is recognized. Based on the recognized context, a context-specific set of sound event classes is selected for the sound event detection stage. The event detection stage also uses context-dependent acoustic models and count-based event priors. Two alternative event detection approaches are studied. In the first one, a monophonic event sequence is outputted by detecting the most prominent sound event at each time instance using Viterbi decoding. The second approach introduces a new method for producing polyphonic event sequence by detecting multiple overlapping sound events using multiple restricted Viterbi passes. A new metric is introduced to evaluate the sound event detection performance with various level of polyphony. This combines the detection accuracy and coarse time-resolution error into one metric, making the comparison of the performance of detection algorithms simpler. The two-step approach was found to improve the results substantially compared to the context-independent baseline system. In the block-level, the detection accuracy can be almost doubled by using the proposed context-dependent event detection.

Normal and Abnormal Non-Speech Audio Event Detection Using MFCC and PR-Based Feature Sets

Non-speech audio event detection and classification has become a very active subject of research, since it can be implemented in many important areas: audio surveillance and context awareness systems. In this study, non-speech normal and abnormal audio events were detected by Mel-frequency cepstrum coefficients (MFCC) and Pitch range (PR) based features using artificial neural network (ANN) classifiers. We have 4 abnormal events (glass breaking, dog barking, scream, gunshot) and 2 normal events (engine noise and rain). Event detection, using ANN classifiers, resulted in an accuracy of up to 92%, with recognition rates overall in the range of 78%-87.5%.

Translation and Capture Architecture for Output of Conversational Utterances

Architecture that combines capture and translation of concepts, goals, needs, locations, objects, locations, and items (e.g., sign text) into complete conversational utterances that take a translation of the item, and morph it with fluidity into sets of sentences that can be echoed to a user, and that the user can select to communicate speech (or textual utterances). A plurality of modalities that process images, audio, video, searches and cultural context, for example, which are representative of at least context and/or content, and can be employed to glean additional information regarding a communications exchange to facilitate more accurate and efficient translation. Gesture recognition can be utilized to enhance input recognition, urgency, and/or emotional interaction, for example. Speech can be used for document annotation. Moreover, translation (e.g., speech to speech, text to speech, speech to text, handwriting to speech, text or audio, . . . ) can be significantly improved in combination with this architecture.

Adaptive Harmonic Spectral Decomposition for Multiple Pitch Estimation

Multiple pitch estimation consists of estimating the fundamental frequencies and saliences of pitched sounds over short time frames of an audio signal. This task forms the basis of several applications in the particular context of musical audio. One approach is to decompose the short-term magnitude spectrum of the signal into a sum of basis spectra representing individual pitches scaled by time-varying amplitudes, using algorithms such as nonnegative matrix factorization (NMF). Prior training of the basis spectra is often infeasible due to the wide range of possible musical instruments. Appropriate spectra must then be adaptively estimated from the data, which may result in limited performance due to overfitting issues. In this paper, we model each basis spectrum as a weighted sum of narrowband spectra representing a few adjacent harmonic partials, thus enforcing harmonicity and spectral smoothness while adapting the spectral envelope to each instrument. We derive a NMF-like algorithm to estimate the model parameters and evaluate it on a database of piano recordings, considering several choices for the narrowband spectra. The proposed algorithm performs similarly to supervised NMF using pre-trained piano spectra but improves pitch estimation performance by 6% to 10% compared to alternative unsupervised NMF algorithms.