Representation Of Speech Research Articles

Understanding Self-Supervised Learning of Speech Representation via Invariance and Redundancy Reduction

Self-supervised learning (SSL) has emerged as a promising paradigm for learning flexible speech representations from unlabeled data. By designing pretext tasks that exploit statistical regularities, SSL models can capture useful representations that are transferable to downstream tasks. Barlow Twins (BTs) is an SSL technique inspired by theories of redundancy reduction in human perception. In downstream tasks, BTs representations accelerate learning and transfer this learning across applications. This study applies BTs to speech data and evaluates the obtained representations on several downstream tasks, showing the applicability of the approach. However, limitations exist in disentangling key explanatory factors, with redundancy reduction and invariance alone being insufficient for factorization of learned latents into modular, compact, and informative codes. Our ablation study isolated gains from invariance constraints, but the gains were context-dependent. Overall, this work substantiates the potential of Barlow Twins for sample-efficient speech encoding. However, challenges remain in achieving fully hierarchical representations. The analysis methodology and insights presented in this paper pave a path for extensions incorporating further inductive priors and perceptual principles to further enhance the BTs self-supervision framework.

LeBenchmark 2.0: A standardized, replicable and enhanced framework for self-supervised representations of French speech

Self-supervised learning (SSL) is at the origin of unprecedented improvements in many different domains including computer vision and natural language processing. Speech processing drastically benefitted from SSL as most of the current domain-related tasks are now being approached with pre-trained models. This work introduces LeBenchmark 2.0 an open-source framework for assessing and building SSL-equipped French speech technologies. It includes documented, large-scale and heterogeneous corpora with up to 14,000 h of heterogeneous speech, ten pre-trained SSL wav2vec 2.0 models containing from 26 million to one billion learnable parameters shared with the community, and an evaluation protocol made of six downstream tasks to complement existing benchmarks. LeBenchmark 2.0 also presents unique perspectives on pre-trained SSL models for speech with the investigation of frozen versus fine-tuned downstream models, task-agnostic versus task-specific pre-trained models as well as a discussion on the carbon footprint of large-scale model training. Overall, the newly introduced models trained on 14,000 h of French speech outperform multilingual and previous LeBenchmark SSL models across the benchmark but also required up to four times more energy for pre-training.

Preserved Gray Matter Volume in the Left Superior Temporal Gyrus Underpins Speech-in-Noise Processing in Middle-Aged Adults.

Neuroanatomical evidence suggests that behavioral speech-in-noise (SiN) perception and the underlying cortical structural network are altered by aging, and these aging-induced changes could be initiated during middle age. However, the mechanism behind the relationship between auditory performance and neural substrates of speech perception in middle-aged individuals remains unclear. In this study, we measured the structural volumes of selected neuroanatomical regions involved in speech and hearing processing to establish their association with speech perception ability in middle-aged adults. Sentence perception in quiet and noisy conditions was behaviorally measured in 2 different age groups: young (20-39 years old) and middle-aged (40-59-year-old) adults. Anatomical magnetic resonance images were taken to assess the gray matter volume of specific parcellated brain areas associated with speech perception. The relationships between these and behavioral auditory performance with age were determined. The middle-aged adults showed poorer speech perception in both quiet and noisy conditions than the young adults. Neuroanatomical data revealed that the normalized gray matter volume in the left superior temporal gyrus, which is closely related to acoustic and phonological processing, is associated with behavioral SiN perception in the middle-aged group. In addition, the normalized gray matter volumes in multiple cortical areas seem to decrease with age. The results indicate that SiN perception in middle-aged adults is closely related to the brain region responsible for lower-level speech processing, which involves the detection and phonemic representation of speech. Nonetheless, the higher-order cortex may also contribute to age-induced changes in auditory performance.

Speaker voice normalization for end-to-end speech translation

Speaker voices exhibit acoustic variation. Our preliminary experiments reveal that normalized voice can significantly improve end-to-end speech translation. To mitigate the negative impact of acoustic voice variation across speakers on speech translation, we propose SVN-ST, a Speaker-Voice-Normalized end-to-end Speech Translation framework. In SVN-ST, we use synthetic speech inputs generated from a Text-to-Speech system to complement raw speech inputs. In order to explore synthetic speech inputs, we introduce two essential components for SVN-ST: an alignment adapter at the encoder side and a normalized speech knowledge distillation module at the decoder side. The former forces the representations of raw speech inputs to be close to those of synthetic (normalized) speech inputs while the latter attempts to guide the translations of raw speech inputs with those yielded from synthetic speech inputs. Two additional losses are also defined to equip with the two components. Experimental results on the MuST-C benchmark dataset demonstrate that SVN-ST outperforms previous state-of-the-art end-to-end non-normalized speech translation systems by 0.4 BLEU and cascaded speech translation systems by 2.3 BLEU. On the Covost 2 testset, SVN-ST also outperforms other normalized speech methods on robustness. Further analyses suggest that our model effectively aligns speech representations from different speakers, enhances robustness, and significantly improves sentence-level translation quality.

The role of vowel and consonant onsets in neural tracking of natural speech

Objective. To investigate how the auditory system processes natural speech, models have been created to relate the electroencephalography (EEG) signal of a person listening to speech to various representations of the speech. Mainly the speech envelope has been used, but also phonetic representations. We investigated to which degree of granularity phonetic representations can be related to the EEG signal. Approach. We used recorded EEG signals from 105 subjects while they listened to fairy tale stories. We utilized speech representations, including onset of any phone, vowel–consonant onsets, broad phonetic class (BPC) onsets, and narrow phonetic class onsets, and related them to EEG using forward modeling and match–mismatch tasks. In forward modeling, we used a linear model to predict EEG from speech representations. In the match–mismatch task, we trained a long short term memory based model to determine which of two candidate speech segments matches with a given EEG segment. Main results. Our results show that vowel–consonant onsets outperform onsets of any phone in both tasks, which suggests that neural tracking of the vowel vs. consonant exists in the EEG to some degree. We also observed that vowel (syllable nucleus) onsets exhibit a more consistent representation in EEG compared to syllable onsets. Significance. Finally, our findings suggest that neural tracking previously thought to be associated with BPCs might actually originate from vowel–consonant onsets rather than the differentiation between different phonetic classes.

A multimodal dynamical variational autoencoder for audiovisual speech representation learning

High-dimensional data such as natural images or speech signals exhibit some form of regularity, preventing their dimensions from varying independently. This suggests that there exists a lower dimensional latent representation from which the high-dimensional observed data were generated. Uncovering the hidden explanatory features of complex data is the goal of representation learning, and deep latent variable generative models have emerged as promising unsupervised approaches. In particular, the variational autoencoder (VAE) which is equipped with both a generative and an inference model allows for the analysis, transformation, and generation of various types of data. Over the past few years, the VAE has been extended to deal with data that are either multimodal or dynamical (i.e., sequential). In this paper, we present a multimodal and dynamical VAE (MDVAE) applied to unsupervised audiovisual speech representation learning. The latent space is structured to dissociate the latent dynamical factors that are shared between the modalities from those that are specific to each modality. A static latent variable is also introduced to encode the information that is constant over time within an audiovisual speech sequence. The model is trained in an unsupervised manner on an audiovisual emotional speech dataset, in two stages. In the first stage, a vector quantized VAE (VQ-VAE) is learned independently for each modality, without temporal modeling. The second stage consists in learning the MDVAE model on the intermediate representation of the VQ-VAEs before quantization. The disentanglement between static versus dynamical and modality-specific versus modality-common information occurs during this second training stage. Extensive experiments are conducted to investigate how audiovisual speech latent factors are encoded in the latent space of MDVAE. These experiments include manipulating audiovisual speech, audiovisual facial image denoising, and audiovisual speech emotion recognition. The results show that MDVAE effectively combines the audio and visual information in its latent space. They also show that the learned static representation of audiovisual speech can be used for emotion recognition with few labeled data, and with better accuracy compared with unimodal baselines and a state-of-the-art supervised model based on an audiovisual transformer architecture.

Features of linguistic expertise of texts of dialogues on anti-corruption cases

The object of the study is dialogue texts with signs of corrupt speech behavior. The subject of the study are marker words, linguistic methods that reveal linguistic signs of a discourse that has a meaningful and semantic focus on receiving or transferring values for the actions of the recipient (inaction), in favor of the giver. The purpose of this work was to conduct a communicative analysis of corrupt speech behavior in the text–dialogue: to accept illegal remuneration for actions (inaction) in the service in order to obtain valuables in the event of an agreement between the specified persons.The main task of the linguistic and forensic examination of anti-corruption cases is to establish the event by its speech representation: it is required to prove that the dialogue is about the transfer of funds. To solve this issue, it is necessary to conduct a communicative analysis of the text-dialogue. The research methodology is related to the communicative analysis of texts-dialogues with signs of corruption content, based on the works of D.L. Karpov, "on the theoretical and practical works of A. N. Baranov, K. I. Brinev, E. N. Galyashina, M. A. Grachev, I. A. Sternin, etc., devoted to cases related to corruption. The scientific novelty of the work lies in the use of a communicative approach within the framework of text-dialogue analysis, logical-semantic and functional-stylistic analysis. The analysis of the expert study of the text-dialogue on anti-corruption cases is presented, the conceptual framework is developed. As a result of the study, it was concluded that the main task of linguistic expertise in anti-corruption cases is to establish an event by its speech representation through the use of a set of semantic and pragmatic techniques of discursive analysis, which allows us to clarify the signs of conversational dialogue that are significant for the legal qualification of speech crimes. When conducting a communicative analysis of a corrupt text, it is important to identify signs of corrupt speech behavior. When analyzing a communicative corruption situation, it is necessary to consider the speech strategies and tactics of the participants in the dialogue, which ultimately allow us to determine the true intentions of the communicants aimed at achieving a communicative goal.

Sensitive Quantification of Cerebellar Speech Abnormalities Using Deep Learning Models.

Objective, sensitive, and meaningful disease assessments are critical to support clinical trials and clinical care. Speech changes are one of the earliest and most evident manifestations of cerebellar ataxias. This work aims to develop models that can accurately identify and quantify clinical signs of ataxic speech. We use convolutional neural networks to capture the motor speech phenotype of cerebellar ataxia based on time and frequency partial derivatives of log-mel spectrogram representations of speech. We train classification models to distinguish patients with ataxia from healthy controls as well as regression models to estimate disease severity. Classification models were able to accurately distinguish healthy controls from individuals with ataxia, including ataxia participants who clinicians rated as having no detectable clinical deficits in speech. Regression models produced accurate estimates of disease severity, were able to measure subclinical signs of ataxia, and captured disease progression over time. Convolutional networks trained on time and frequency partial derivatives of the speech signal can detect sub-clinical speech changes in ataxias and sensitively measure disease change over time. Learned speech analysis models have the potential to aid early detection of disease signs in ataxias and provide sensitive, low-burden assessment tools in support of clinical trials and neurological care.

Varieties of ironic objection in popular science Internet media text

The article focuses on the objectionable potential of irony using popular science media texts. The empirical basis was the hypermedia text of the news about the refutation of the serotonin theory of depression, which predetermined a significant shift in ideas about the disease. The purpose of the article is to identify and analyze the variability in the implementation of ironic objection. The latter seems to be a complexly organized speech action, which is based on goal setting, which constitutes a subgoal in achieving the main communicative goal of the hypermedia text. The pre-established multitasking of the study determined the choice of approach to analysis — the praxeological method of media linguistics. As the analysis of the material has shown, objection demonstrates the moment of transition of scientific knowledge from hypotheticality to a state of epistemological certainty. Objection helps to overcome unproductive tendencies in the system of academic views. Linguistically significant characteristics of ironic objection in popular science media discourse include extensive heuristic potential, projection of the development of epistemological modality, critical rethinking of extra-linguistic situations, involvement of the addressee in discussion, debatable nature, and requiring interpretive effort. The results obtained during the study — linguistic models of the representation of ironic objection — demonstrate a paradigm shift in scientific views about the mechanisms of depressive disorder and the key lines of understanding in the public consciousness of the problems correlated with the identified epistemological crisis. In this study, patterns of speech representation of ironic objection are identified and typologized on the basis of repeated and regularly reproduced linguistic markers.

ZMM-TTS: Zero-Shot Multilingual and Multispeaker Speech Synthesis Conditioned on Self-Supervised Discrete Speech Representations

ZMM-TTS: Zero-Shot Multilingual and Multispeaker Speech Synthesis Conditioned on Self-Supervised Discrete Speech Representations

Research on the Innovation of University English Teaching Mode Driven by Artificial Intelligence

Abstract With the continuous deepening and development of information technology in the field of education, the combination of artificial intelligence technology and university English teaching has become a trend. This paper constructs a human-machine symbiotic English teaching model based on intelligent speech recognition, which takes into account university English learning abilities and learning objectives. Specifically, the multi-scale attention mechanism is used instead of the Conformer model to extract the global information of speech features and the local information of different scales and fuse them to learn more speech representations of different scales so as to better recognize the teacher and student behaviors in the English classroom. Meanwhile, combining the teaching mechanism of human-computer symbiosis, we construct a human-computer symbiosis English teaching model for speech recognition in an intelligent classroom and receive feedback after teaching experiments. The occupancy rate of teacher and student behaviors in the English classroom is 50.86% and 49.14%, respectively, and the conversion rate of teacher and student behaviors is 29.78%, which achieves a high level of teacher-student interaction in this model. In addition, the class utilizing the English teaching model of this paper was 5.02 points higher than the traditional class after one semester of teaching, P=0.038<0.05, and the results showed a significant difference. Through the comparison of cases and teaching experiments, the reasonableness of the English teaching model based on speech recognition is demonstrated, and the research content and problem areas of this paper are summarized.

QS-TTS: Towards Semi-Supervised Text-to-Speech Synthesis via Vector-Quantized Self-Supervised Speech Representation Learning

QS-TTS: Towards Semi-Supervised Text-to-Speech Synthesis via Vector-Quantized Self-Supervised Speech Representation Learning

Adapting Pre-Trained Self-Supervised Learning Model for Speech Recognition with Light-Weight Adapters

Self-supervised learning (SSL) is an effective way of learning rich and transferable speech representations from unlabeled data to benefit downstream tasks. However, effectively incorporating a pre-trained SSL model into an automatic speech recognition (ASR) system remains challenging. In this paper, we propose a network architecture with light-weight adapters to adapt a pre-trained SSL model for an end-to-end (E2E) ASR. An adapter is introduced in each SSL network layer and trained on the downstream ASR task, while the parameters of the pre-trained SSL network layers remain unchanged. By carrying over all pre-trained parameters, we avoid the catastrophic forgetting problem. At the same time, we allow the network to quickly adapt to ASR task with light-weight adapters. The experiments using LibriSpeech and Wall Street Journal (WSJ) datasets show that (1) the proposed adapter-based fine-tuning consistently outperforms full-fledged training in low-resource scenarios, with up to 17.5%/12.2% relative word error rate (WER) reduction on the 10 min LibriSpeech split; (2) the adapter-based adaptation also shows competitive performance in high-resource scenarios, which further validates the effectiveness of the adapters.

VatLM: Visual-Audio-Text Pre-Training With Unified Masked Prediction for Speech Representation Learning

Although speech is a simple and effective way for humans to communicate with the outside world, a more realistic speech interaction contains multimodal information, e.g., vision, text. How to design a unified framework to integrate different modal information and leverage different resources (e.g., visual-audio pairs, audio-text pairs, unlabeled speech, and unlabeled text) to facilitate speech representation learning was not well explored. In this paper, we propose a unified cross-modal representation learning framework VATLM (Visual-Audio-Text Language Model). The proposed VATLM employs a unified backbone network to model the modality-independent information and utilizes three simple modality-dependent modules to preprocess visual, speech, and text inputs. In order to integrate these three modalities into one shared semantic space, VATLM is optimized with a masked prediction task of unified tokens, given by our proposed unified tokenizer. We evaluate the pre-trained VATLM on audio-visual related downstream tasks, including audio-visual speech recognition (AVSR), visual speech recognition (VSR) tasks. Results show that the proposed VATLM outperforms previous the state-of-the-art models, such as audio-visual pre-trained AV-HuBERT model, and analysis also demonstrates that VATLM is capable of aligning different modalities into the same space. To facilitate future research, we release the code and pre-trained models at https://aka.ms/vatlm.

Gender Role and Hate Speech Representation in Web Series

Abstract: In India, media has been an essential component of life. The effect of the media can be seen with this that majority of people across the globe rely on various media platforms to increase consciousness and get information for enjoyment and to gain knowledge about a variety of other global events. Denis McQuail (2000) believes that “the media is accountable for illuminating and fostering systematic exposure, as well as influencing ideas and ideals.” Furthermore, according to the BARC (Broadcast Audience Research Council) Report, 2019, “web series are defined as popular shows with high viewing that cover a wide range of socio-cultural elements of people's life.” There are several types of streaming platforms for entertainment that broadcast 24*7. India much like other jurisdictions has been struggling with the issue of how to control hate speech on the internet. Regulating internet hate speech is a complex issue. The aim of this study is to examine web series in the perspective of Indian society. The stereotypical images of women and hate speech, including abusive language and rude behavior, and discrimination on the basis of religion and caste, have been portrayed in Indian web series. The current study intends to investigate how the Indian web series content structure continues to impact gender role and hate speech representation on streaming platforms.

Dialect in the Making

Within the framework of third-wave sociolinguistic research, Asif Agha’s (2003) theory of enregisterment has proved a successful approach to explore the mechanisms that lead to the indexical connection between language and identity. Beal (2009, 2020), Cooper (2013, 2020), and Ruano-García (2012, 2020, 2021), among others, have investigated this phenomenon from a diachronic perspective. They have highlighted the value of dialect writing as a window into the main features associated with particular dialects, as it draws upon authenticating practices such as the use of dialect respellings, which not only signal salient phonological features, but also link them to wider schemes of sociocultural values and identities. This paper seeks to add to this field of research by looking at literary representations of Derbyshire speech (1850–1900) through the lens of enregisterment. My aims are twofold: I attempt to (1) shed light on the main phonological features of the Derbyshire dialect, while (2) determining how this variety was enregistered in the Late Modern English period, and whether meaningful text type-dependent indexical shifts might have affected the way in which this dialect was understood and thus represented by native and non-native speakers.

Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision

Abstract We introduce SPEAR-TTS, a multi-speaker text-to-speech (TTS) system that can be trained with minimal supervision. By combining two types of discrete speech representations, we cast TTS as a composition of two sequence-to-sequence tasks: from text to high-level semantic tokens (akin to “reading”) and from semantic tokens to low-level acoustic tokens (“speaking”). Decoupling these two tasks enables training of the “speaking” module using abundant audio-only data, and unlocks the highly efficient combination of pretraining and backtranslation to reduce the need for parallel data when training the “reading” component. To control the speaker identity, we adopt example prompting, which allows SPEAR-TTS to generalize to unseen speakers using only a short sample of 3 seconds, without any explicit speaker representation or speaker labels. Our experiments demonstrate that SPEAR-TTS achieves a character error rate that is competitive with state-of-the-art methods using only 15 minutes of parallel data, while matching ground-truth speech in naturalness and acoustic quality.

Improving speech emotion recognition by fusing self-supervised learning and spectral features via mixture of experts

Speech Emotion Recognition (SER) is an important area of research in speech processing that aims to identify and classify emotional states conveyed through speech signals. Recent studies have shown considerable performance in SER by exploiting deep contextualized speech representations from self-supervised learning (SSL) models. However, SSL models pre-trained on clean speech data may not perform well on emotional speech data due to the domain shift problem. To address this problem, this paper proposes a novel approach that simultaneously exploits an SSL model and a domain-agnostic spectral feature (SF) through the Mixture of Experts (MoE) technique. The proposed approach achieves the state-of-the-art performance on weighted accuracy compared to other methods in the IEMOCAP dataset. Moreover, this paper demonstrates the existence of the domain shift problem of SSL models in the SER task.

Applying the Lombard Effect to Speech-in-Noise Communication

This study explored how the Lombard effect, a natural or artificial increase in speech loudness in noisy environments, can improve speech-in-noise communication. This study consisted of several experiments that measured the impact of different types of noise on synthesizing the Lombard effect. The main steps were as follows: first, a dataset of speech samples with and without the Lombard effect was collected in a controlled setting; then, the frequency changes in the speech signals were detected using the McAulay and Quartieri algorithm based on a 2D speech representation; next, an average formant track error was computed as a metric to evaluate the quality of the speech signals in noise. Three image assessment methods, namely the SSIM (Structural SIMilarity) index, RMSE (Root Mean Square Error), and dHash (Difference Hash) were used for this purpose. Furthermore, this study analyzed various spectral features of the speech signals in relation to the Lombard effect and the noise types. Finally, this study proposed a method for automatic noise profiling and applied pitch modifications to neutral speech signals according to the profile and the frequency change patterns. This study used an overlap-add synthesis in the STRAIGHT vocoder to generate the synthesized speech.

Interpreting Convolutional Layers in DNN Model Based on Time–Frequency Representation of Emotional Speech

Abstract The paper describes the relations of speech signal representation in the layers of the convolutional neural network. Using activation maps determined by the Grad-CAM algorithm, energy distribution in the time–frequency space and their relationship with prosodic properties of the considered emotional utterances have been analysed. After preliminary experiments with the expressive speech classification task, we have selected the CQT-96 time–frequency representation. Also, we have used a custom CNN architecture with three convolutional layers in the main experimental phase of the study. Based on the performed analysis, we show the relationship between activation levels and changes in the voiced parts of the fundamental frequency trajectories. As a result, the relationships between the individual activation maps, energy distribution, and fundamental frequency trajectories for six emotional states were described. The results show that the convolutional neural network in the learning process uses similar fragments from time–frequency representation, which are also related to the prosodic properties of emotional speech utterances. We also analysed the relations of the obtained activation maps with time-domain envelopes. It allowed observing the importance of the speech signals energy in classifying individual emotional states. Finally, we compared the energy distribution of the CQT representation in relation to the regions’ energy overlapping with masks of individual emotional states. In the result, we obtained information on the variability of energy distributions in the selected signal representation speech for particular emotions.