Speaker Information Research Articles

Effective Zero-Shot Multi-Speaker Text-to-Speech Technique Using Information Perturbation and a Speaker Encoder

Effective Zero-Shot Multi-Speaker Text-to-Speech Technique Using Information Perturbation and a Speaker Encoder

Role of discursive structure in the genre specification of online discourses

The paper starts with the assumption that each time people engage in discourses, they perform social actions in order to adaptively satisfy their communicative purposes and needs (cf. Verschueren–Brisard 2009; Steen 2011; Giltrow 2013). Moreover, these social actions form types in accordance with the typical co-occurrences of typical discourse situations involving typical roles, typical themes and typical communicative goals tar- geted by discourse participants (cf. Bakhtin 1986). These typical co-occurrences serve as ground for the functioning of genre knowledge and result in typical patterns of construal.
 In this paper, I present a two-pronged empirical study that reveals the role of discursive structure in the creation of the genre-specific character of online discourses. By discursive structural units, I mean discourse segments (usually larger than one sentence but smaller than the entire discourse) used by the speaker to accomplish a communicative strategy aimed at achieving a more specific communicative purpose. During the research, I used two types of data. On the one hand, I compiled a research corpus consisting of 50 online recipes and 50 online book reviews in Hungarian. On the other hand, I conducted an experiment in which one group of Hungarian native speaker informants (25 persons) had to create a recipe for a good novel, while another group of informants (25 persons) had to write a review of a given recipe. The research thus focused on the genres of recipe and (book) review. I performed corpus-based qualitative analyses on both datasets using the MAXQDA software. After carefully developing a data-driven annotation system on a smaller pilot material, I systematically identified the communicative strategic units of both the non-elicited and the elicited dataset [cf. Swales’s (1990) move structure analysis]. The research findings are as follows: (i) Discursive structure formed by the speaker’s communicative strategies is a genre-marking quality of the online discourses under study. (ii) Recipes written by laypersons and professionals do not have a different discursive structure but the identified discursive structure is very clearly manifested. (iii) Book reviews written by lay readers differ significantly from those written by professional critics in terms of discursive structure and this sharp differentiation reveals a separation between two more specific genres, i.e. amateur book review and professional critique.

Speaker diarization with variants of self-attention and joint speaker embedding extractor

The speaker diarization task pertains to the automated differentiation of speakers within an audio recording, while lacking any prior information regarding the speakers. The introduction of the self-attention mechanism in End-to-End Neural Speaker Diarization (EEND) has elegantly resolved the issue of overlapping speakers. The Transformer model equipped with self-attention mechanism has shown great potential in collecting global information, yielding remarkable outcomes in various tasks. However, the individual speaker characteristics are predominantly reflected in the contextual information, which conventional self-attention would not adequately address. In this study, we propose a hierarchical encoders model to augment the encoders’ acquisition of speaker information in two distinct ways: (1) Constraining the perceptual field of the self-attentive mechanism with left-right windows or Gaussian weights to highlight contextual information; (2) Utilizing a pre-trained time-delay neural network based speaker embedding extractor to alleviate the shortcomings of speaker feature extraction ability. We evaluate the proposed methods on a simulated dataset of two speakers and a real conversation dataset. The model with the most favorable outcomes among the proposed enhancements achieves a diarization error rate of 7.74% on the simulated dataset and 21.92% on MagicData-RAMC after adaptation. These results compellingly demonstrate the efficacy of the proposed methods.

W2VC: WavLM representation based one-shot voice conversion with gradient reversal distillation and CTC supervision

Non-parallel data voice conversion (VC) has achieved considerable breakthroughs due to self-supervised pre-trained representation (SSPR) being used in recent years. Features extracted by the pre-trained model are expected to contain more content information. However, in common VC with SSPR, there is no special implementation to remove speaker information in the content representation extraction by SSPR, which prevents further purification of the speaker information from SSPR representation. Moreover, in conventional VC, Mel-spectrogram is often selected as the reconstructed acoustic feature, which is not consistent with the input of the content encoder and results in some information lost. Motivated by the above, we proposed W2VC to settle the issues. W2VC consists of three parts: (1) We reconstruct feature from WavLM representation (WLMR) that is more consistent with the input of content encoder; (2) Connectionist temporal classification (CTC) is used to align content representation and text context from phoneme level, content encoder plus gradient reversal layer (GRL) based speaker classifier are used to remove speaker information in the content representation extraction; (3) WLMR-based HiFi-GAN is trained to convert WLMR to waveform speech. VC experimental results show that GRL can purify well the content information of the self-supervised model. The GRL purification and CTC supervision on the content encoder are complementary in improving the VC performance. Moreover, the synthesized speech using the WLMR retrained vocoder achieves better results in both subjective and objective evaluation. The proposed method is evaluated on the VCTK and CMU databases. It is shown the method achieves 8.901 in objective MCD, 4.45 in speech naturalness, and 3.62 in speaker similarity of subjective MOS score, which is superior to the baseline.

MHG-ERC: Multi-hypergraph Feature Aggregation Network for Emotion Recognition in Conversations

The modeling of conversational context is an essential step in Emotion Recognition in Conversations (ERC). To maintain high performance and a low GPU memory consumption, this article proposes a new idea of using multiple hypergraphs to model the conversational context and designs a multi-hypergraph feature aggregation network for ERC. We use context window, speaker information, position information between utterances, and specific step size to construct different hyperedges. Then, various hypergraphs generated by different hyperedges are used to aggregate local and remote context information in turn. Experiments on two dialogue emotion datasets, IEMOCAP and MELD, demonstrate the effectiveness and superiority of this new model. In addition, our model requires only relatively low GPU memory consumption.

Speaking Race or Racialized Speaking: Evidence from Perceptions of Lateral Variants by Puerto Rican Listeners

Abstract: In this study we show that the perception of lateral variants by Puerto Rican listeners changes according to who the listener believes is speaking. Puerto Rican listeners heard sentences with target words featuring either rhotic [ɾ] or lateral [l] (amo[ɾ] – amo[l]) codas, a sociophonetic alternation that is common in Puerto Rican Spanish. Across four conditions characterized by the presence or absence of social cues, listeners judged speakers on five Likert items: honesty, intelligence, SES, work ethic, and pleasantness. Condition 1 included only the sociophonetic variant while conditions 2, 3 and 4 included information about speaker nationality (flag, either Puerto Rican or Dominican), or racial phenotype (pictures of people) or both nationality and racial phenotype. The results revealed that for the sociophonetic variant only condition (no additional speaker information), listeners rated the lateral variant more negatively. However, for the three conditions where speaker information was provided, ratings were significantly affected by listeners’ perception of racial phenotype and nationality, with almost no interaction with phonetic variant. These results suggest that what was a significant effect for sociophonetic variant (condition 1) was overshadowed by perceptions of speaker racial phenotype and nationality. In other words, racialized speaking matters.

Deep Learning Enabled Semantic Communications With Speech Recognition and Synthesis

In this paper, we develop a deep learning based semantic communication system for speech transmission, named DeepSC-ST. We take the speech recognition and speech synthesis as the transmission tasks of the communication system, respectively. First, the speech recognition-related semantic features are extracted for transmission by a joint semantic-channel encoder and the text is recovered at the receiver based on the received semantic features, which significantly reduces the required amount of data transmission without performance degradation. Then, we perform speech synthesis at the receiver, which dedicates to re-generate the speech signals by feeding the recognized text and the speaker information into a neural network module. To enable the DeepSC-ST adaptive to dynamic channel environments, we identify a robust model to cope with different channel conditions. According to the simulation results, the proposed DeepSC-ST significantly outperforms conventional communication systems and existing DL-enabled communication systems, especially in the low signal-to-noise ratio (SNR) regime. A software demonstration is further developed as a proof-of-concept of the DeepSC-ST.

Avoiding dominance of speaker features in speech-based depression detection

The performance of speech-based depression detectors is limited by the scarcity and imbalance in depression data. We found that depression detectors could be strongly biased toward speaker features when the number of training speakers is insufficient. To address this issue, we propose a speaker-invariant depression detector (SIDD) that minimizes speaker information in the latent space. The SIDD consists of an autoencoder, a depression classifier, and a speaker-embedding projector. By incorporating speaker-embedding vectors into the autoencoder’s latent vectors, speaker information is effectively eliminated for the depression classifier. Experimental results demonstrate significant improvements achieved by minimizing speaker information, and our proposed method generally outperforms previous approaches for depression detection on the DAIC-WOZ dataset.

Low-Resource Language Information Processing using Dwarf Mongoose Optimization with Deep Learning Based Sentiment Classification

Asian and low-resource language information processing refers to the field of computational linguistics that aims to develop natural language processing (NLP) technologies for languages that have fewer available language resources or are less commonly spoken. This is an important field of study because many languages in Asia and other parts of the world are underrepresented in the field of NLP, which may limit access to information and technology for speakers of these languages. The growing volume of user-generated content on the web has made sentiment analysis (SA) a significant tool for extracting data regarding human emotional states. Twitter sentiment detectors provide a superior solution for assessing the quality of products and services compared to other conventional technologies. The detection performance and classifier accuracy of SA, which can be highly dependent on classifier methods and the quality of input features have been utilised. Deep learning (DL) methods use distinct techniques to extract data from raw data such as tweets or texts and represent them in different forms of models. Therefore, this article presents a Dwarf Mongoose Optimization with Deep Learning-Based Twitter Sentiment Classification (DMODL-TSC) technique to classify sentiments based on tweets. The presented DMODL-TSC technique leverages the concepts of natural language processing (NLP) and DL. Primarily, the raw tweets are preprocessed to transform them into a useful format. Next, the DMODL-TSC technique uses the advanced FastText word embedding technique. Moreover, the bidirectional recurrent neural network (BiRNN) method is utilized for the recognition of sentiments. Finally, the DMO technique is utilized for the optimal hyperparameter optimization of the BiRNN method, which leads to effective classification performance. The comprehensive result examination of the DMODL-TSC system was tested on three datasets, and the obtained outcomes illustrate the supremacy of the DMODL-TSC approach.

Space-and-speaker-aware acoustic modeling with effective data augmentation for recognition of multi-array conversational speech

We propose a space-and-speaker-aware (SSA) approach to acoustic modeling (AM), denoted as SSA-AM, to improve system performances of automatic speech recognition (ASR) in distant multi-array conversational scenarios. In contrast to conventional AM which only uses spectral features from a target speaker as inputs, the inputs to SSA-AM consists of speech features from both the target and interfering speakers, which contain discriminative information from different speakers, including spatial information embedded in interaural phase differences (IPDs) between individual interfering speakers and the target speaker. In the proposed SSA-AM framework, we explore four acoustic model architectures consisting of different combinations of four neural networks, namely deep residual network, factorized time delay neural network, self-attention and residual bidirectional long short-term memory neural network. Various data augmentation techniques are adopted to expand the training data to include different options of beamformed speech obtained from multi-channel speech enhancement. Evaluated on the recent CHiME-6 Challenge Track 1, our proposed SSA-AM framework achieves consistent recognition performance improvements when compared with the official baseline acoustic models. Furthermore, SSA-AM outperforms acoustic models without explicitly using the space and speaker information. Finally, our data augmentation schemes are shown to be especially effective for compact model designs. Code is released at https://github.com/coalboss/SSA_AM.

What do end-to-end speech models learn about speaker, language and channel information? A layer-wise and neuron-level analysis

Deep neural networks are inherently opaque and challenging to interpret. Unlike hand-crafted feature-based models, we struggle to comprehend the concepts learned and how they interact within these models. This understanding is crucial not only for debugging purposes but also for ensuring fairness in ethical decision-making. In our study, we conduct a post-hoc functional interpretability analysis of pretrained speech models using the probing framework (Hupkes et al., 2018). Specifically, we analyze utterance-level representations of speech models trained for various tasks such as speaker recognition and dialect identification. We conduct layer and neuron-wise analyses, probing for speaker, language, and channel properties. Our study aims to answer the following questions: (i) what information is captured within the representations? (ii) how is it represented and distributed? and (iii) can we identify a minimal subset of the network that possesses this information? Our results reveal several novel findings, including: (i) channel and gender information are distributed across the network, (ii) the information is redundantly available in neurons with respect to a task, (iii) complex properties such as dialectal information are encoded only in the task-oriented pretrained network, (iv) and is localized in the upper layers, (v) we can extract a minimal subset of neurons encoding the pre-defined property, (vi) salient neurons are sometimes shared between properties, (vii) our analysis highlights the presence of biases (for example gender) in the network. Our cross-architectural comparison indicates that: (i) the pretrained models capture speaker-invariant information, and (ii) CNN models are competitive with Transformer models in encoding various understudied properties.

Facilitating Access to Current, Evidence-Based Health Information for Non-English Speakers.

Scientific communication is crucial for the development of societies and the advancement of knowledge. However, many countries, and, consequently, their researchers, clinicians and community members, lack access to this information due to the information being disseminated in English rather than their native language. In this viewpoint, we aim to discuss the impacts of this problem and also outline recommendations for facilitating non-English speakers' access to current, evidence-based health information, thus extending the impact of science beyond academia. First, the authors discuss the barriers to accessing scientific health information for non-English speakers and highlight the negative impact of imposing English as a predominant language in academia. Next, the authors discuss the impacts of reduced access to clinical information for non-English speakers and how this reduced access impacts clinicians, clients, and health systems. Finally, the authors provide recommendations for enhancing access to scientific communication worldwide.

Identity Retention and Emotion Converted StarGAN for low-resource emotional speaker recognition

In speaker recognition (SR) task, the inconsistency of emotion between enrolled speech and test speech will lead to emotion domain mismatch, which decreases the recognition accuracy and is more serious under low-resource training data condition. To help SR model learn the various emotion distribution when only neutral speech is enrolled, Identity Retention and Emotion Converted StarGAN (IREC-StarGAN) is proposed to generate speech feature in other emotions. Emotion and speaker relevant multi-task loss is designed to retain origin speaker information during adversarial training progress for the generation of emotion feature. To avoid essential speaker or emotion information being contaminated by irrelevant regions, generator uses SC-bottleneck with multi kernel size channels to learn the deep feature of data in various scales. Discriminator adopts densenet with two branches, one of which classifies whether input feature belongs to target emotion domain and the another one focuses on the speaker information. The origin neutral feature will be enrolled with generated feature together to be recognized by Resnet18 for speaker recognition task. The proposed framework obtains the EER of 18.33% on Mandarin Affective Speech Corpus and 8.14% on IEMOCAP, which outperform the existing embedding and GAN methods and reveals the robustness of IREC-StarGAN under low-resource emotion condition.

Mel-S3R: Combining Mel-spectrogram and self-supervised speech representation with VQ-VAE for any-to-any voice conversion

The self-supervised speech representation (S3R) has succeeded in many downstream tasks, such as speaker recognition and voice conversion thanks to its high-level information. Voice conversion (VC) is a task to convert the source speech into a target speaker’s voice. Though S3R features effectively encode content and speaker information, spectral features contain low-level acoustic information that is complementary to the S3R. As a result, solely relying on the S3R features for VC may not be optimal. In order to seek speech representation carrying both high-level learned information and low-level spectral details for VC, we proposed a three-level attention to combine Mel-spectrogram (Mel) and S3R, denoted as Mel-S3R. In particular, S3R features are high-level learned representations extracted by a pre-trained network with self-supervised learning. Whereas Mel is the spectral feature representing the acoustic information. Then the proposed Mel-S3R is used as the input of any-to-any VQ-VAE-based VC and the experiments are performed as a downstream task. Objective metrics and subjective listening tests have demonstrated that the proposed Mel-S3R speech representation facilitates the VC framework to achieve robust performance in terms of both speech quality and speaker similarity.

Comparing the comprehension of short stories paired with familiar and unfamiliar melodies and spoken text in autistic children

Autistic children (AC) can demonstrate varying abilities of comprehending spoken and written information. Although information is often paired with music to facilitate recall and learning, there is limited research investigating if sung short stories influence the listening and reading comprehension (LRC) of AC. The purpose of this study was to compare the LRC of information paired with (1) familiar melodies, (2) unfamiliar melodies, and (3) spoken information in AC. Participants ( N = 59) received three different short stories presented live that were sung paired with a familiar melody, sung paired with an unfamiliar melody, and spoken. Both music conditions had guitar accompaniment. After presentation of each condition, participants answered five close-ended questions to measure LRC. There was no significant within-group LRC difference between the familiar melody, unfamiliar melody, or spoken conditions. Descriptive statistics indicated that both music conditions yielded slightly higher mean LRC scores than the spoken condition with slightly higher mean LRC during the familiar melody condition than the unfamiliar melody condition. Given the differences in the means, future researchers should be mindful of the possibility of a type II error. Implications for classroom and clinical work, limitations of the study, and suggestions for future research are provided.

Joint Syntax-Enhanced and Topic-Driven Graph Networks for Emotion Recognition in Multi-Speaker Conversations

Daily conversations contain rich emotional information, and identifying this emotional information has become a hot task in the field of natural language processing. The traditional dialogue sentiment analysis method studies one-to-one dialogues and cannot be effectively applied to multi-speaker dialogues. This paper focuses on the relationship between participants in a multi-speaker conversation and analyzes the influence of each speaker on the emotion of the whole conversation. We summarize the challenges of emotion recognition work in multi-speaker dialogue, focusing on the context-topic switching problem caused by multi-speaker dialogue due to its free flow of topics. For this challenge, this paper proposes a graph network that combines syntactic structure and topic information. A syntax module is designed to convert sentences into graphs, using edges to represent dependencies between words, solving the colloquial problem of daily conversations. We use graph convolutional networks to extract the implicit meaning of discourse. In addition, we focus on the impact of topic information on sentiment, so we design a topic module to optimize the topic extraction and classification of sentences by VAE. Then, we use the combination of attention mechanism and syntactic structure to strengthen the model’s ability to analyze sentences. In addition, the topic segmentation technology is adopted to solve the long-term dependencies problem, and a heterogeneous graph is used to model the dialogue. The nodes of the graph combine speaker information and utterance information. Aiming at the interaction relationship between the subject and the object of the dialogue, different edge types are used to represent different interaction relationships, and different weights are assigned to them. The experimental results of our work on multiple public datasets show that the new model outperforms several other alternative methods in sentiment label classification results. In the multi-person dialogue dataset, the classification accuracy is increased by more than 4%, which verifies the effectiveness of constructing heterogeneous dialogue graphs.

Valency Types in South Ethiosemitic Languages

This article analyses the main characteristics of verb valency types and the behaviour of verbs in morphosyntactic structures in South Ethiosemitic languages. To achieve the objective, native speaker informants, written texts, and introspection (for Amharic) were used as sources of data. The findings reveal that six valency types of verbs are in use in the languages: avalent, monovalent, bivalent, trivalent, and a valency of ambivalent verbs with one or two arguments, as well as intransitive verbs with a cognate object. Except for the type that concerns avalent verbs, the rest are highly productive. The valency types were differentiated by the required clause elements that precede the verb (e.g., direct object, indirect object, or locative adverbial). Except for the avalent type, all the rest may include obligatory subject(s) and, in some cases, optional adjunct(s).

ZSE-VITS: A Zero-Shot Expressive Voice Cloning Method Based on VITS

Voice cloning aims to synthesize the voice with a new speaker’s timbre from a small amount of the new speaker’s speech. Current voice cloning methods, which focus on modeling speaker timbre, can synthesize speech with similar speaker timbres. However, the prosody of these methods is flat, lacking expressiveness and the ability to control the expressiveness of cloned speech. To solve this problem, we propose a novel method ZSE-VITS (zero-shot expressive VITS) based on the end-to-end speech synthesis model VITS. Specifically, we use VITS as the backbone network and add the speaker recognition model TitaNet as the speaker encoder to realize zero-shot voice cloning. We use explicit prosody information to avoid effects from the speaker information and adjust speech prosody using the prosody information prediction and prosody fusion methods directly. We widen the pitch distribution of the train datasets using pitch augmentation to improve the generalization ability of the prosody model, and we fine-tune the prosody predictor alone in the emotion corpus to learn prosody prediction of various styles. The objective and subjective evaluations of the open datasets show that our method can generate more expressive speech and adjust prosody information artificially without affecting the similarity of speaker timbre.

Dialogue Relation Extraction with Document-Level Heterogeneous Graph Attention Networks

We propose a heterogeneous graph attention network to address the problem of dialogue relation extraction. Compared with several popular sequence-based and graph-based models, our method shows superior performance on the benchmark dataset DialogRE. The implementation of this work can be found at https://github.com/declare-lab/dialog-HGAT Dialogue relation extraction aims to detect the relation between pairs of entities mentioned in a multi-party dialogue. It plays an essential role in understanding the deep logic of dialogues and facilitating the development of intelligent dialogue systems. We introduce a heterogeneous graph attention network to model the cross-sentence relations in a conversation. This heterogeneous graph attention network has modeled multi-type features of the conversation, such as utterance, word, speaker, argument, and entity type information. We compare our method with several popular baselines such as convolutional neural networks and long short-term memory, experimental results show our model outperforms the state-of-the-art method by 9.4%/7.8% F1 scores, and 6.6%/3.9% $$F1_c$$ scores in both validation and test sets with only 4.0M parameters. In this work, we present an attention-based heterogeneous graph network to deal with the dialogue relation extraction task in an inductive manner. Experimental results on the dataset DialogRE confirm the effectiveness of our method.

Speech-Section Extraction Using Lip Movement and Voice Information in Japanese

In recent years, several Japanese companies have attempted to improve the efficiency of their meetings, which has been a significant challenge. For instance, voice recognition technology is used to considerably improve meeting minutes creation. In an automatic minutes-creating system, identifying the speaker to add speaker information to the text would substantially improve the overall efficiency of the process. Therefore, a few companies and research groups have proposed speaker estimation methods; however, it includes challenges, such as requiring advance preparation, special equipment, and multiple microphones. These problems can be solved by using speech sections that are extracted from lip movements and voice information. When a person speaks, voice and lip movements occur simultaneously. Therefore, the speaker’s speech section can be extracted from videos by using lip movement and voice information. However, when this speech section contains only voice information, the voiceprint information of each meeting participant is required for speaker identification. When using lip movements, the speech section and speaker position can be extracted without the voiceprint information. Therefore, in this study, we propose a speech-section extraction method that uses image and voice information in Japanese for speaker identification. The proposed method consists of three processes: i) the extraction of speech frames using lip movements, ii) the extraction of speech frames using voices, and iii) the classification of speech sections using these extraction results. We used video data to evaluate the functionality of the method. Further, the proposed method was compared with state-of-the-art techniques. The average F-measure of the proposed method is determined to be higher than that of the conventional methods that are based on state-of-the-art techniques. The evaluation results showed that the proposed method achieves state-of-the-art performance using a simpler process compared to the conventional method.