Multiple Speakers Research Articles

GLGAN-VC: A Guided Loss-Based Generative Adversarial Network for Many-to-Many Voice Conversion.

Many-to-many voice conversion (VC) is a technique aimed at mapping speech features between multiple speakers during training and transferring the vocal characteristics of one source speaker to another target speaker, all while maintaining the content of the source speech unchanged. Existing research highlights a notable gap between the original and generated speech samples in terms of naturalness within many-to-many VC. Therefore, there is substantial room for improvement in achieving more natural-sounding speech samples for both parallel and nonparallel VC scenarios. In this study, we introduce a generative adversarial network (GAN) system with a guided loss (GLGAN-VC) designed to enhance many-to-many VC by focusing on architectural improvements and the integration of alternative loss functions. Our approach includes a pair-wise downsampling and upsampling (PDU) generator network for effective speech feature mapping (FM) in multidomain VC. In addition, we incorporate an FM loss to preserve content information and a residual connection (RC)-based discriminator network to improve learning. A guided loss (GL) function is introduced to efficiently capture differences in latent feature representations between source and target speakers, and an enhanced reconstruction loss is proposed for better contextual information preservation. We evaluate our model on various datasets, including VCC 2016, VCC 2018, VCC 2020, and an emotional speech dataset (ESD). Our results, based on both subjective and objective evaluation metrics, demonstrate that our model outperforms state-of-the-art (SOTA) many-to-many GAN-based VC models in terms of speech quality and speaker similarity in the generated speech samples.

Training-Based Multiple Source Tracking Using Manifold-Learning and Recursive Expectation-Maximization

In this paper we propose a data-driven approach for multiple speaker tracking in reverberant enclosures. The speakers are uttering, possibly overlapping, speech signals while moving in the environment. The method comprises two stages. The first stage executes a single source localization using semi-supervised learning on multiple manifolds. The second stage, which is unsupervised, uses time-varying maximum likelihood estimation for tracking. The feature vectors, used by both stages, are the relative transfer functions (RTFs), which are known to be related to source positions. The number of sources is assumed to be known while the microphone positions are unknown. In the training stage, a large database of RTFs is given. A small percentage of the data is attributed with exact positions (namely, labelled data) and the rest is assumed to be unlabelled, i.e. the respective position is unknown. Then, a nonlinear, manifold-based, mapping function between the RTFs and the source positions is inferred. Applying this mapping function to all unlabelled RTFs constructs a dense grid of localized sources. In the test phase, this RTFs grid serves as the centroids for a (MoG) model. The MoG parameters are estimated by applying a recursive variant of the (EM) procedure that relies on the sparsity and intermittency of the speech signals. We present a comprehensive simulation study in various reverberation levels, including static and dynamic scenarios, for both two or three (partially) overlapping speakers. For the dynamic case we provide simulations with several speakers trajectories, including intersecting sources. The proposed scheme outperforms baseline methods that use a simpler propagation model in terms of localization accuracy and tracking capabilities.

Segment-Less Continuous Speech Separation of Meetings: Training and Evaluation Criteria

Continuous Speech Separation (CSS) has been proposed to address speech overlaps during the analysis of realistic meeting-like conversations by eliminating any overlaps before further processing. CSS separates a recording of arbitrarily many speakers into a small number of overlap-free output channels, where each output channel may contain speech of multiple speakers. Often, a separation model is trained with Utterance-level Permutation Invariant Training (uPIT), which exclusively maps a speaker to an output channel, and applied in a sliding window approach called stitching. Recently, we introduced an alternative training scheme called Graph-PIT that teaches the separator to produce a speaker-shared output channel format without stitching. It can handle an arbitrary number of speakers as long as the number of overlapping speakers is never larger than the number of output channels. Models trained in this way are able to perform segment-less CSS, i.e., without stitching, and achieve comparable and often better separation quality than the conventional CSS with uPIT and stitching. In this contribution, we further investigate the Graph-PIT training scheme. We show in extended experiments that Graph-PIT also works in challenging reverberant conditions. We simplify the training schedule for Graph-PIT with the recently proposed Source Aggregated Signal-to-Distortion Ratio (SA-SDR) loss, which eliminates unfavorable properties of the previously used A-SDR loss to enable training with Graph-PIT from scratch. Furthermore, we introduce novel signal-level evaluation metrics for meeting scenarios, namely the source-aggregated scale- and convolution-invariant Signal-to-Distortion Ratio (SA-SI-SDR and SA-CI-SDR), which are generalizations of the commonly used SDR-based metrics for the CSS case.

Speaker-Independent Emotional Voice Conversion via Disentangled Representations

Emotional Voice Conversion (EVC) technology aims to transfer emotional state in speech while keeping the linguistic information and speaker identity unchanged. Prior studies on EVC have been limited to perform the conversion for a specific speaker or a predefined set of multiple speakers seen in the training stage. When encountering arbitrary speakers that may be unseen during training (outside the set of speakers used in training), existing EVC methods have limited conversion capabilities. However, converting the emotion of arbitrary speakers, even those unseen during the training procedure, in one model is much more challenging and much more attractive in real-world scenarios. To address this problem, in this study, we propose SIEVC, a novel speaker-independent emotional voice conversion framework for arbitrary speakers via disentangled representation learning. The proposed method employs the autoencoder framework to disentangle the emotion information and emotion-independent information of each input speech into separated representation spaces. To achieve better disentanglement, we incorporate mutual information minimization into the training process. In addition, adversarial training is applied to enhance the quality of the generated audio signals. Finally, speaker-independent EVC for arbitrary speakers could be achieved by only replacing the emotion representations of source speech with the target ones. The experimental results demonstrate that the proposed EVC model outperforms the baseline models in terms of objective and subjective evaluation for both seen and unseen speakers.

A voice-driven embodied interaction system with response actions according to speech speed by the mora

Focusing on the correlation between speech sounds and body movements, we developed a voice-driven embodied entrainment character called InterActor that automatically generates communicative motions from speech, and demonstrated the effectiveness of the system. We also developed a communication agent that responds appropriately to utterance contents by focusing on words in speech using speech recognition. However, these systems simply generate body motions from the burst-pause of speech. The response actions corresponding to speaker’s speech speeds were not considered. In this paper, we analyze the relationship between an individual’s speech activity and speech speed[mora/s] from various speech experiments with the aim of facilitating speech according to speaker’s speech characteristics. We develop a voice-driven embodied entrainment system which performs nodding with different speeds according to speech speeds based on speech recognition. Based on the results of the speech experiments, we conduct an online evaluation experiment using videos of multiple speakers with different speaking speeds using the robot as a platform to verify the effectiveness of the response actions generated by the system.

Predicting Level-Dependent Changes in Concurrent Vowel Scores Using the 2D-CNN Models

Differences in fundamental frequencies (F0s) are an important cue for segregating multiple speakers. However, the ability to avail this cue for identification varies with sound levels. For different-and same-F0 conditions, the identification scores of both vowels increased from low- to mid-levels and then reduced at higher levels for younger adults with normal hearing (YNH). These subjects benefited from the F0 difference; however, this benefit varied across the levels. The current study aims to develop a deep-neural-network (DNN) model that can predict the level-dependent changes in concurrent-vowel scores for YNH subjects. This DNN-model includes two-dimensional convolutional neural networks (2D-CNNs) in parallel, with the same architecture, to predict the concurrent-vowel scores. The input layer was neural responses of the auditory-nerve model to concurrent vowels. The 2D-CNN models were trained and validated using the subsets of the concurrent vowels from 50, 65, and 75 dB SPL for both F0 conditions, using the batch gradient descent algorithm. The trained model was then fine-tuned with a single epoch. The 2D-CNN models were evaluated against vowel levels (25 to 85 dB SPL) and F0 conditions. Compared with previous models, the current model accurately predicts the level-dependent changes in concurrent vowel scores for different-and same-F0 conditions.

Dialogue Logic Aware and Key Utterance Decoupling Model for Multi-Party Dialogue Reading Comprehension

Multi-party dialogue machine reading comprehension (MRC) brings an unprecedented challenge due to the multiple speakers and the complex discourse linkages among speaker-aware utterances. The majority of current methods only consider the textual aspects of dialogue situations, and pay little attention to crucial speaker-aware cues. This prevents a model from capturing the speaker’s intention and important discourse information for questions in a complex discourse relationship, leading to the model giving wrong answers. In this paper, we construct a dialogue logic graph module by the relational graph convolutional network (R-GCN) to structure the dialogue information, and design a speaker prediction task to enhance the ability to capture discourse logic. Additionally, we construct a key utterance information decoupling module that focuses on the key discourse information flow involve questions, and filters out noise information. Extensive experiments FriendsQA and Molweni show that our approach outperforms competitive baselines and current state-of-the-art models, especially when dealing with more rounds of dialogue and questions involving people, events and time.

VStyclone: Real-time Chinese voice style clone

This paper proposes a novel Chinese speech cloning model named VStyclone, which consists of three stages: multi-speaker training, target speaker encoding, and target speaker synthesis. In this work, we design an efficient tone extractor, which can reallocate resources to the sequences of log-mel spectrogram frames obtained from multiple speakers’ speech, thus allowing the network to learn multiple speakers’ features differently. This approach allows the network to focus more on the voice features of the target speaker and extract the target features accurately. To cluster the voices of the same speaker and sparse the voices of different speakers, we build an optimal softmax loss to optimize the model. Then, we develop a style synthesizer, which adopts the idea of transformer instead of recurrent neural network, so that the model can not only process text information in parallel, but also improve the model's ability to process long-distance contextual information. Meanwhile, we embed a style extraction module in the style synthesizer to dynamically capture style ranges in an unsupervised manner. In addition, the VStyclone model uses generative adversarial networks as the base generation model of the vocoder to improve the generation speed, which runs 1.2 times faster than the real-time generation speed on CPU, and finally the VStyclone model obtains the SOTA effect.

Simultaneous Execution of Dereverberation, Denoising, and Speaker Separation Using a Neural Beamformer for Adapting Robots to Real Environments

It remains challenging for robots to accurately perform sound source localization and speech recognition in a real environment with reverberation, noise, and the voices of multiple speakers. Accordingly, we propose “U-TasNet-Beam,” a speech extraction method for extracting only the target speaker’s voice from all ambient sounds in a real environment. U-TasNet-Beam is a neural beamformer comprising three elements: a neural network for removing reverberation and noise, a second neural network for separating the voices of multiple speakers, and a minimum variance distortionless response (MVDR) beamformer. Experiments with simulated data and recorded data show that the proposed U-TasNet-Beam can improve the accuracy of sound source localization and speech recognition in robots compared to the conventional methods in a noisy, reverberant, and multi-speaker environment. In addition, we propose the spatial correlation matrix loss (SCM loss) as a loss function for the neural network learning the spatial information of the sound. By using the SCM loss, we can improve the speech extraction performance of the neural beamformer.

Interactivity/Reciprocity (Online Discussions/Discussion Quality)

Interactivity/Reciprocity (Online Discussions/Discussion Quality)

Arm motion symmetry in conversation

Data-driven synthesis of human motion during conversational speech is an active research area with applications that include character animation, computer gaming and conversational agents. Natural looking motion is key to both perceived realism and understanding of any synthesised animation. Multi-modal speech and body-motion data is scarce and limited, so it is common to augment real motion data by mirroring the body pose to double the number of training samples. This augmentation is based on the assumption that a person’s gesturing is not affected by handedness and that the reflected pose is plausible. In this study, we explore the validity of this assumption by evaluating the reflective symmetry of a speaker’s arms during conversational exchanges. We analyse the left and right arm motion of 36 subjects during dyadic conversation and present the per-frame symmetry of the arm gestures. To identify temporal offsets caused by the presence of a leading hand, we compute the time lag between movements of the left and right arms. We perform a nearest neighbour search to test the validity of any mirrored pose. We also consider information theory to examine the information gain from mirroring the data. We implement a speech-to-gesture generative model to determine the efficacy of lateral mirroring techniques for data augmentation. Our findings suggest that both positional symmetry and left–right motion offsets vary from speaker to speaker. We conclude that data augmentation by mirroring is valid in certain cases when considering the mirrored pose as a new virtual identity, but that it should be carefully considered as a generic approach if the gesturing style and handedness of the original speaker is to be maintained. • Review the motion symmetry of multiple speakers during dyadic conversation. • Analyse positional, temporal and informational symmetry of arm motion. • Discuss the efficacy of lateral mirroring of the human body for data augmentation. • Conclude lateral mirroring is not suited as a generic approach. • Suggest lateral mirroring as a new identity is a suitable data augmentation method. • Propose our statistical analysis for evaluating speech-driven conversational agents.

Human–machine collaboration in transcription

As automatic speech recognition (ASR) has improved, it has become a viable tool for content transcription. Prior to the use of ASR for this task, content transcription was achieved through human effort alone. Despite improvements, ASR performance is as yet imperfect, especially in more challenging conditions (eg multiple speakers, noise, nonstandard accents). Given this, a promising way forward is a human-in-the-loop (HIL) approach. This contribution describes our work with HIL ASR on the transcription task. Traditionally, ASR performance has been measured using word error rate (WER). This measure may not be sufficient to describe the full set of errors that a speech-to-text (STT) pipeline designed for transcription can make, such as those involving capitalisation, punctuation, and inverse text normalisation (ITN). It is therefore the case that improved WER does not always lead to increased productivity, and the inclusion of ASR in HIL may adversely affect productivity if it contains too many errors. Rev.com provides a convenient laboratory to explore these questions. Originally, the company provided transcriptions of audio and video content executed solely by humans (known as Revvers). More recently, ASR was introduced in an HIL workflow where Revvers postedited an ASR first draft. We provide an analysis of the interaction between metrics of ASR accuracy and the productivity of our 72,000+ Revvers transcribing more than 15,000 hours of media every week. To do this, we utilise two measures of transcriptionist productivity: transcriber real time factor (RTF) and words per minute (WPM). Through our work, we hope to focus attention on the human productivity and quality of experience (QoE) aspects of improvements in ASR and related technologies. Given the broad scope of content transcription applications and the still elusive objective of perfect machine performance, keeping the human in the loop in both practice and mind is critical. This paper provides an overview of human and machine transcription and Rev’s marketplace, followed by an analysis of the relationship between ASR accuracy and transcriptionist productivity, and concludes with suggestions for future work.

Improving Few-Shot Multi-Speaker Text-to-Speech Adaptive-Based with Extracting Mel-Vector (EMV) for Vietnamese

Training a multi-speaker Text-to-Speech (TTS) model requires multiple speakers’ voices to generate an average speech model. However, the average speech synthesis model will be distorted or averaged, resulting in low quality if the new speaker’s voice has too little data to train. The existing methods require fine-tuning the model; otherwise, the model will achieve low adaptive quality. However, for synthesis voice to achieve high adaptive quality, at least thousands of fine-tuning steps are required. To solve these issues, in this paper, we propose a Vietnamese multi-speaker TTS adaptive-based technique that synthesizes high-quality speech and effectively adapts to new speakers, with two main improvements: (1) propose an Extracting Mel-Vector (EMV) architecture with three components, the Encoder–Decoder–Embedding Features, which enables complete learning of speaker features with Mel-spectrograms as input for few-shot training and (2) a continuous-learning technique called “data-distributing” preserves the new speaker’s characteristics after many training epochs. Our proposed model outperformed the baseline multi-speaker synthesis model and achieved a MOS score of 3.8/4.6 and SIM of 2.6/4 with only 1 min of the target speaker’s voice.

Speaker Adaptation on Articulation and Acoustics for Articulation-to-Speech Synthesis.

Silent speech interfaces (SSIs) convert non-audio bio-signals, such as articulatory movement, to speech. This technology has the potential to recover the speech ability of individuals who have lost their voice but can still articulate (e.g., laryngectomees). Articulation-to-speech (ATS) synthesis is an algorithm design of SSI that has the advantages of easy-implementation and low-latency, and therefore is becoming more popular. Current ATS studies focus on speaker-dependent (SD) models to avoid large variations of articulatory patterns and acoustic features across speakers. However, these designs are limited by the small data size from individual speakers. Speaker adaptation designs that include multiple speakers’ data have the potential to address the issue of limited data size from single speakers; however, few prior studies have investigated their performance in ATS. In this paper, we investigated speaker adaptation on both the input articulation and the output acoustic signals (with or without direct inclusion of data from test speakers) using the publicly available electromagnetic articulatory (EMA) dataset. We used Procrustes matching and voice conversion for articulation and voice adaptation, respectively. The performance of the ATS models was measured objectively by the mel-cepstral distortions (MCDs). The synthetic speech samples were generated and are provided in the supplementary material. The results demonstrated the improvement brought by both Procrustes matching and voice conversion on speaker-independent ATS. With the direct inclusion of target speaker data in the training process, the speaker-adaptive ATS achieved a comparable performance to speaker-dependent ATS. To our knowledge, this is the first study that has demonstrated that speaker-adaptive ATS can achieve a non-statistically different performance to speaker-dependent ATS.

Cortical tracking of voice pitch in the presence of multiple speakers depends on selective attention.

Voice pitch carries linguistic and non-linguistic information. Previous studies have described cortical tracking of voice pitch in clean speech, with responses reflecting both pitch strength and pitch value. However, pitch is also a powerful cue for auditory stream segregation, especially when competing streams have pitch differing in fundamental frequency, as is the case when multiple speakers talk simultaneously. We therefore investigated how cortical speech pitch tracking is affected in the presence of a second, task-irrelevant speaker. We analyzed human magnetoencephalography (MEG) responses to continuous narrative speech, presented either as a single talker in a quiet background or as a two-talker mixture of a male and a female speaker. In clean speech, voice pitch was associated with a right-dominant response, peaking at a latency of around 100 ms, consistent with previous electroencephalography and electrocorticography results. The response tracked both the presence of pitch and the relative value of the speaker’s fundamental frequency. In the two-talker mixture, the pitch of the attended speaker was tracked bilaterally, regardless of whether or not there was simultaneously present pitch in the speech of the irrelevant speaker. Pitch tracking for the irrelevant speaker was reduced: only the right hemisphere still significantly tracked pitch of the unattended speaker, and only during intervals in which no pitch was present in the attended talker’s speech. Taken together, these results suggest that pitch-based segregation of multiple speakers, at least as measured by macroscopic cortical tracking, is not entirely automatic but strongly dependent on selective attention.

Conversations as Vessels and Anchors

Conversations as Vessels and Anchors

A Time-Frequency Bins Selection Pipeline for Direction-of-Arrival Estimation Using a Single Acoustic Vector Sensor

The case that multiple speakers are simultaneously active is usually regarded as adverse condition in the context of multi-source direction-of-arrival (DOA) estimation. Moreover, in adjacent sources scenarios, classifying DOA estimates by directions may merge adjacent sources and misclassify reverberation as sources, i.e., there is a mismatch between sources and reverberation. In order to address the open challenges mentioned above, this work introduces a pipeline for selecting reliable time-frequency (TF) bins and estimating multi-source DOAs using a single acoustic vector sensor (AVS). In the proposed pipeline, we show a viewpoint that the multiple simultaneously active sources condition can be exploited to select TF bins solely dominated by sources. The consistency of the distribution of intensity vectors (IVs) in multi-source scenarios is derived mathematically to support the proposed viewpoint. We address the mismatch between sources and reverberations in adjacent sources scenarios by focusing only on the direction with the highest intensity vector density (IVD), regardless of how many sources dominate the given TF window. Simulation and experimental results show that the proposed method can accurately and robustly estimate the multi-source DOAs in various scenarios.

Cross-lingual multi-speaker speech synthesis with limited bilingual training data

Modeling voices for multiple speakers and multiple languages with one speech synthesis system has been a challenge for a long time, especially in low-resource cases. This paper presents two approaches to achieve cross-lingual multi-speaker text-to-speech (TTS) and code-switching synthesis under two training scenarios: (1) cross-lingual synthesis with sufficient data, (2) cross-lingual synthesis with limited data per speaker. Accordingly, a novel TTS synthesis model and a non-autoregressive multi-speaker voice conversion model are proposed. The TTS model designed for sufficient-data cases has a Tacotron-based structure that uses shared phonemic representations associated with numeric language ID codes. As for the data-limited scenario, we adopt a framework cascading several speech modules to achieve our goal. In particular, we proposed a non-autoregressive many-to-many voice conversion module to address multi-speaker synthesis for data-insufficient cases. Experimental results on speaker similarity show that our proposed voice conversion module can maintain the voice characteristics well in data-limited cases. Both approaches use limited bilingual data and demonstrate impressive performance in cross-lingual synthesis, which can deliver fluent foreign speech and even code-switching speech for monolingual speakers.

Conversational emotion recognition studies based on graph convolutional neural networks and a dependent syntactic analysis

Multimodal Emotion Recognition for Conversation (ERC) is a challenging multi-class classification task that requires recognizing multiple speakers’ emotions in text, audio, video, and other modalities. ERC has received considerable attention from researchers due to its potential applications in opinion mining, advertising, and healthcare. However, the syntactic structure characteristics of the text itself have not been considered in this study. Taking into account this, this paper proposes a conversational affective analysis model (DSAGCN) combining dependent syntactic analysis and graph convolutional neural networks. Since words that reflect emotional polarity are usually concentrated exclusively in limited regions, the DSAGCN model first employs a self-attention mechanism to capture the most effective words in the dialogue context and obtain a more accurate vector representation of the emotional semantics. Then, based on speaker relationships and dependent syntactic relationships, the multimodal sentiment relationship graphs are constructed. Finally, a graph convolutional neural network is used to complete the recognition of multimodal emotion. In extensive experiments on two real datasets, IEMOCAP and MELD, the DSAGCN model outperforms the existing models in terms of average accuracy and f1 values for multimodal emotion recognition, especially for emotions such as “happiness” and “anger”. Thus, dependent syntactic analysis and self-attention mechanism can enhance the model’s ability to understand emotions.

DOA-guided source separation with direction-based initialization and time annotations using complex angular central Gaussian mixture models

By means of spatial clustering and time-frequency masking, a mixture of multiple speakers and noise can be separated into the underlying signal components. The parameters of a model, such as a complex angular central Gaussian mixture model (cACGMM), can be determined based on the given signal mixture itself. Then, no misfit between training and testing conditions arises, as opposed to approaches that require labeled datasets to be trained. Whereas the separation can be performed in a completely unsupervised way, it may be beneficial to take advantage of a priori knowledge. The parameter estimation is sensitive to the initialization, and it is necessary to address the frequency permutation problem. In this paper, we therefore consider three techniques to overcome these limitations using direction of arrival (DOA) estimates. First, we propose an initialization with simple DOA-based masks. Secondly, we derive speaker specific time annotations from the same masks in order to constrain the cACGMM. Thirdly, we employ an approach where the mixture components are specific to each DOA instead of each speaker. We conduct experiments with sudden DOA changes, as well as a gradually moving speaker. The results demonstrate that particularly the DOA-based initialization is effective to overcome both of the described limitations. In this case, even methods based on normally unavailable oracle information are not observed to be more beneficial to the permutation resolution or the initialization. Lastly, we also show that the proposed DOA-guided source separation works quite robustly in the presence of adverse conditions and realistic DOA estimation errors.