Speaker Information Research Articles

Optimization of the area under the ROC curve using neural network supervectors for text-dependent speaker verification

This paper explores two techniques to improve the performance of text-dependent speaker verification systems based on deep neural networks. Firstly, we propose a general alignment mechanism to keep the temporal structure of each phrase and obtain a supervector with the speaker and phrase information, since both are relevant for a text-dependent verification. As we show, it is possible to use different alignment techniques to replace the global average pooling providing significant gains in performance. Moreover, we also present a novel Back-end approach to train a neural network for detection tasks by optimizing the Area Under the Curve (AUC) as an alternative to the usual triplet loss function, so the system is end-to-end, with a cost function close to our desired measure of performance. As we can see in the experimental section, this approach improves the system performance, since our triplet neural network based on an approximation of the AUC (aAUC) learns how to discriminate between pairs of examples from the same identity and pairs of different identities. The different alignment techniques to produce supervectors in addition to the new Back-end approach were tested on the RSR2015-Part I and RSR2015-Part II database for text-dependent speaker verification, providing competitive results compared to similar size networks using the global average pooling to extract supervectors and using a simple Back-end or triplet loss training.

The distribution of speaker information in Dutch fricatives /s/ and /x/ from telephone dialogues.

Although previous work has shown that some speech sounds are more speaker-specific than others, not much is known about the speaker information of the same segment in different linguistic contexts. The present study, therefore, investigated whether Dutch fricatives /s/ and /x/ from telephone dialogues contain differential speaker information as a function of syllabic position and labial co-articulation. These linguistic effects, established in earlier work on read broadband speech, were first investigated. Using a corpus of Dutch telephone speech, results showed that the telephone bandwidth captures the expected effects of perseverative and anticipatory labialization for dorsal fricative /x/, for which spectral peaks fall within the telephone band, but not for coronal fricative /s/, for which the spectral peak falls outside the telephone band. Multinomial logistic regression shows that /s/ contains slightly more speaker information than /x/ in telephone speech and that speaker information is distributed across the speech signal in a systematic way; even though differences in classification accuracy were small, codas and tokens with labial neighbors yielded higher scores than onsets and tokens with non-labial neighbors for both /s/ and /x/. These findings indicate that speaker information in the same speech sound is not the same across linguistic contexts.

Non-Parallel Sequence-to-Sequence Voice Conversion With Disentangled Linguistic and Speaker Representations

This article presents a method of sequence-to-sequence (seq2seq) voice conversion using non-parallel training data. In this method, disentangled linguistic and speaker representations are extracted from acoustic features, and voice conversion is achieved by preserving the linguistic representations of source utterances while replacing the speaker representations with the target ones. Our model is built under the framework of encoder-decoder neural networks. A recognition encoder is designed to learn the disentangled linguistic representations with two strategies. First, phoneme transcriptions of training data are introduced to provide the references for leaning linguistic representations of audio signals. Second, an adversarial training strategy is employed to further wipe out speaker information from the linguistic representations. Meanwhile, speaker representations are extracted from audio signals by a speaker encoder. The model parameters are estimated by two-stage training, including a pre-training stage using a multi-speaker dataset and a fine-tuning stage using the dataset of a specific conversion pair. Since both the recognition encoder and the decoder for recovering acoustic features are seq2seq neural networks, there are no constrains of frame alignment and frame-by-frame conversion in our proposed method. Experimental results showed that our method obtained higher similarity and naturalness than the best non-parallel voice conversion method in Voice Conversion Challenge 2018. Besides, the performance of our proposed method was closed to the state-of-the-art parallel seq2seq voice conversion method.

A Study of Inductive Biases for Unsupervised Speech Representation Learning

Distributed representations, or embeddings, are commonly learned without supervision on very large unannotated corpora for natural language processing. In speech processing, deep network-based representations such as bottlenecks and x-vectors have had some success,but are limited to supervised or partly supervised settings where annotations are available and are not optimized to separate underlying factors. Here, we propose a generative model with deep encoders and decoders that can learn interpretable speech representations without supervision. Our inductive biases operate as prior distributions in a variational autoencoder model and allow us to separate several latent variables along a continuous range of time-scale properties, as opposed to binary oppositions or hierarchical factorization that have been previously proposed. On simulated data, we confirm that these biases enable the model to accurately recover phonetic and speaker underlying factors. On TIMIT and LibriSpeech, they yield representations that separate phonetic and speaker information, as evidenced by unsupervised results on downstream phoneme and speaker classification tasks using a simple k-means classifier.

The Role of Lexical Tone Information in the Recognition of Mandarin Sentences in Listeners With Hearing Aids.

Lexical tone information provides redundant cues for the recognition of Mandarin sentences in listeners with normal hearing in quiet conditions. The contribution of lexical tones to Mandarin sentence recognition in listeners with hearing aids (HAs) is unclear. This study aimed to remove lexical tone information and examine the effects on Mandarin sentence intelligibility in HA users. The second objective was to investigate the contribution of cognitive abilities (i.e., general cognitive ability, working memory, and attention) on Mandarin sentence perception when the presentation of lexical tone information was mismatched. A text-to-speech synthesis engine was used to manipulate Mandarin sentences into three test conditions: (1) a Normal Tone test condition, where no alterations were made to lexical tones within sentences; (2) a Flat Tone test condition, where lexical tones were all changed into tone 1 (i.e., the flat tone); and (3) a Random Tone test condition, where each word in test sentences was randomly assigned one of four Mandarin lexical tones. The manipulated sentence signals were presented to 32 listeners with HAs in both quiet and noisy environments at an 8 dB signal to noise ratio. Speech intelligibility was reduced significantly (by approximately 40 percentage points) in the presence of mismatched lexical tone information in both quiet and noise. The difficulty in correctly identifying sentences with mismatched lexical tones among adults with hearing loss was significantly greater than that of adults with normal hearing. Cognitive function was not significantly related to a decline in speech recognition scores. Contextual and other phonemic cues (i.e., consonants and vowels) are inadequate for HA users to perceive sentences with mismatched lexical tone contours in quiet or noise. Also, HA users with better cognitive function could not compensate for the loss of lexical tone information. These results highlight the importance of accurately representing lexical tone information for Mandarin speakers using HAs.

One-Shot Voice Conversion Algorithm Based on Representations Separation

Voice Conversion (VC) is a method of converting the source speaker's speech into the target speaker's speech without changing the source speaker's speech content. The current VC methods have the following problems: (1) they are only applicable to a limited number of speakers, not to any speakers, as a result, the application scenarios are greatly restricted; (2) the representation (feature) separation(RS) effect of the current mainstream technology is not ideal on the source speaker speech and the target speaker speech; and (3) the voice conversion quality of most models is unsatisfactory, and hence needs to be improved. Therefore, in this paper, we constructed a one-shot VC model of Representation Separation, called RS-VC model, implemented by the encoder-decoder structure. The encoder is composed of a content encoder and a speaker encoder. The content encoder separates the content information of the source speaker voice and generates a content representation. The speaker encoder separates the target speaker information of the target speaker voice and generates a speaker representation. The decoder synthesizes the content representation and the speaker representation to generate the converted voice. In this paper, we obtained the optimized speaker verification model SVIGEN2E (Speaker Verification with Instance Normalization using Generalized End-to-End loss) by improving the speaker verification (SV) model. The model SVIGEN2E is used as the speaker encoder. This speaker encoder needs to be trained in advance prior to RS-VC model training, and the pre-trained model of SVINGE2E directly extracts speaker representation of the target speaker's voice, and is used for training and testing RS-VC model. A progressive training method is proposed then for training RS-VC model. Experiments show that the progressive training method can effectively improve the quality of the converted voice. Compared with the basic speaker verification model, both SVINGE2E and RS-VC deliver the impressive improvements in EER (Equal Error Rate).

Variational Domain Adversarial Learning With Mutual Information Maximization for Speaker Verification

Domain mismatch is a common problem in speaker verification (SV) and often causes performance degradation. For the system relying on the Gaussian PLDA backend to suppress the channel variability, the performance would be further limited if there is no Gaussianity constraint on the learned embeddings. This paper proposes an information-maximized variational domain adversarial neural network (InfoVDANN) that incorporates an InfoVAE into domain adversarial training (DAT) to reduce domain mismatch and simultaneously meet the Gaussianity requirement of the PLDA backend. Specifically, DAT is applied to produce speaker discriminative and domain-invariant features, while the InfoVAE performs variational regularization on the embedded features so that they follow a Gaussian distribution. Another benefit of the InfoVAE is that it avoids posterior collapse in VAEs by preserving the mutual information between the embedded features and the training set so that extra speaker information can be retained in the features. Experiments on both SRE16 and SRE18-CMN2 show that the InfoVDANN outperforms the recent VDANN, which suggests that increasing the mutual information between the embedded features and input features enables the InfoVDANN to extract extra speaker information that is otherwise not possible.

ConvS2S-VC: Fully Convolutional Sequence-to-Sequence Voice Conversion

This paper proposes a voice conversion (VC) method using sequence-to-sequence (seq2seq or S2S) learning, which flexibly converts not only the voice characteristics but also the pitch contour and duration of input speech. The proposed method, called ConvS2S-VC, has three key features. First, it uses a model with a fully convolutional architecture. This is particularly advantageous in that it is suitable for parallel computations using GPUs. It is also beneficial since it enables effective normalization techniques such as batch normalization to be used for all the hidden layers in the networks. Second, it achieves many-to-many conversion by simultaneously learning mappings among multiple speakers using only a single model instead of separately learning mappings between each speaker pair using a different model. This enables the model to fully utilize available training data collected from multiple speakers by capturing common latent features that can be shared across different speakers. Owing to this structure, our model works reasonably well even without source speaker information, thus making it able to handle any-to-many conversion tasks. Third, we introduce a mechanism, called the conditional batch normalization that switches batch normalization layers in accordance with the target speaker. This particular mechanism has been found to be extremely effective for our many-to-many conversion model. We conducted speaker identity conversion experiments and found that ConvS2S-VC obtained higher sound quality and speaker similarity than baseline methods. We also found from audio examples that it could perform well in various tasks including emotional expression conversion, electrolaryngeal speech enhancement, and English accent conversion.

Word Reordering for Translation into Korean Sign Language Using Syntactically-guided Classification

Machine translation aims to break the language barrier that prevents communication with others and increase access to information. Deaf people face huge language barriers in their daily lives, including access to digital and spoken information. There are very few digital resources for sign language processing. In this article, we present a transfer-based machine translation system for translating Korean-to-Korean Sign Language (KSL) glosses, mainly composed of (1) dictionary-based lexical transfer and (2) a hybrid syntactic transfer based on a data-driven model. In particular, we formulate complicated word reordering problems in syntactic transfer as multi-class classification tasks and propose “syntactically guided” data-driven syntactic transfer. The core part of our study is a neural classification model for reordering order-important constituent pairs with a reordering task that is newly designed for Korean-to-KSL translation. The experiment results evaluated on news transcript data show that the proposed system achieves a BLEU score of 0.512 and a RIBES score of 0.425, significantly improving upon the baseline system performance.

Gender Prediction by Speech Analysis

Speech is one of the important methods of communication for humans. The speech signal itself contain linguistic information that can be used to identify the speaker information such as gender, emotions and many more. There are some problems that involve in detection gender of the speaker. In forensic analysis, the police need to detect criminal profile from any evidence such as voice from any calls and while in healthcare aspect, some vocal fold pathologies can be bias to a particular gender such as vocal fold cyst can be seen particularly in female patients and the patient will have problem with their voices. Three features are extract from the speech signal which are Mel Frequency Cepstrum Coefficient (MFCC), Linear Prediction Coding (LPC) and Linear Prediction Coding Coefficient (LPCC). While for the classification, two classifier are used which are Support Vector Machine (SVM) and k-Nearest Neighbour (KNN). The recognition rate is higher for the combination of MFCC and LPCC compared to other features. SVM classifier had outperformed KNN classifier and obtained highest recognition rate of 97.45%. Lastly a graphical user interface system is develop that will record the voice of the speakers, pre-process the signal, extract MFCC and LPCC and classify it using SVM.

Speaker-Informed time-and-Content-Aware attention for spoken language understanding

To mitigate the ambiguity of spoken language understanding (SLU) of an utterance, we propose contextual models that can consider the relevant context by using temporal and content-related information effectively. We first propose two axes: ‘Awareness’ and ‘Attention Level’. Awareness includes three methods that consider the timing or content-similarity of context. The Attention Level includes three methods that consider speaker roles to calculate the importance of each historic utterance. By combining one method from each axis, we build various contextual models. The proposed models are designed to use a dataset to automatically learn the importance of previous utterances in terms of time and content. We also propose various speaker information that would be helpful to improve SLU accuracy. The proposed models achieved state-of-the-art F1 scores in experiments on the Dialog State Tracking Challenge (DSTC) 4 and Loqui benchmark datasets. We applied in-depth analysis to verify that the proposed methods are effective to improve SLU accuracy. The analysis also demonstrated the effectiveness of the proposed methods.

Audio Word2vec: Sequence-to-Sequence Autoencoding for Unsupervised Learning of Audio Segmentation and Representation

In text, word2vec transforms each word into a fixed-size vector used as the basic component in applications of natural language processing. Given a large collection of unannotated audio, audio word2vec can also be trained in an unsupervised way using a sequence-to-sequence autoencoder (SA). These vector representations are shown to effectively describe the sequential phonetic structures of the audio segments. In this paper, we further extend this research in the following two directions. First, we disentangle phonetic information and speaker information from the SA vector representations. Second, we extend audio word2vec from the word level to the utterance level by proposing a new segmental audio word2vec in which unsupervised spoken word boundary segmentation and audio word2vec are jointly learned and mutually enhanced, and utterances are directly represented as sequences of vectors carrying phonetic information. This is achieved by means of a segmental sequence-to-sequence autoencoder, in which a segmentation gate trained with reinforcement learning is inserted in the encoder.

Short-term accommodation as a function of addressee language proficiency

AbstractThis paper examines patterns of phonetic accommodation as a function of addressee target language proficiency. Specifically, it analyzes short-term adjustments in the articulation of coda consonants /s/, /ɾ/, and /n/ in the speech of eight New York Dominican Spanish speakers during a series of conversations with different addressees – a native speaker and three nonnative Spanish speakers who have varying levels of Spanish proficiency. Results demonstrate that addressee native-speaker status and proficiency play a statistically significant role in both the degree and direction of phonetic accommodation exhibited by the native speaker informants. While the informants converge with both the most- and least-proficient addressees, they initially diverge from the mid-proficient addressee. The study finds that the native speaker informants use overtly-prestigious variants to attune to the academic Spanish of the most-proficient addressees and use covertly-prestigious, emblematic variants with both the mid-proficient and native speaker addressees to demonstrate outgroup and ingroup membership, respectively.

Supervector Extraction for Encoding Speaker and Phrase Information with Neural Networks for Text-Dependent Speaker Verification

In this paper, we propose a new differentiable neural network with an alignment mechanism for text-dependent speaker verification. Unlike previous works, we do not extract the embedding of an utterance from the global average pooling of the temporal dimension. Our system replaces this reduction mechanism by a phonetic phrase alignment model to keep the temporal structure of each phrase since the phonetic information is relevant in the verification task. Moreover, we can apply a convolutional neural network as front-end, and, thanks to the alignment process being differentiable, we can train the network to produce a supervector for each utterance that will be discriminative to the speaker and the phrase simultaneously. This choice has the advantage that the supervector encodes the phrase and speaker information providing good performance in text-dependent speaker verification tasks. The verification process is performed using a basic similarity metric. The new model using alignment to produce supervectors was evaluated on the RSR2015-Part I database, providing competitive results compared to similar size networks that make use of the global average pooling to extract embeddings. Furthermore, we also evaluated this proposal on the RSR2015-Part II. To our knowledge, this system achieves the best published results obtained on this second part.

SpeakerBeam: Speaker Aware Neural Network for Target Speaker Extraction in Speech Mixtures

The processing of speech corrupted by interfering overlapping speakers is one of the challenging problems with regards to today's automatic speech recognition systems. Recently, approaches based on deep learning have made great progress toward solving this problem. Most of these approaches tackle the problem as speech separation, i.e., they blindly recover all the speakers from the mixture. In some scenarios, such as smart personal devices, we may however be interested in recovering one target speaker from a mixture. In this paper, we introduce SpeakerBeam, a method for extracting a target speaker from the mixture based on an adaptation utterance spoken by the target speaker. Formulating the problem as speaker extraction avoids certain issues such as label permutation and the need to determine the number of speakers in the mixture. With SpeakerBeam, we jointly learn to extract a representation from the adaptation utterance characterizing the target speaker and to use this representation to extract the speaker. We explore several ways to do this, mostly inspired by speaker adaptation in acoustic models for automatic speech recognition. We evaluate the performance on the widely used WSJ0-2mix and WSJ0-3mix datasets, and these datasets modified with more noise or more realistic overlapping patterns. We further analyze the learned behavior by exploring the speaker representations and assessing the effect of the length of the adaptation data. The results show the benefit of including speaker information in the processing and the effectiveness of the proposed method.

A study of English-Chinese Simultaneous Interpreting in Conference on Women Rights Under Chernov’s Compression Theory

Simultaneous interpreting (SI) means the conveying of information of speakers by interpreters fluently and simultaneously with no long pauses. According to Gile’s theory (1995), interpreters have to possess three abilities, including listening and analysis, production and memory. Given the extreme situation of SI, interpreters must have a nice command of processing strategies to ensure a good delivery. And one of the most important strategies is compression proposed by Chernov (2004). To Chernov’s mind, compression is divided into syllabic compression, lexical compression, semantic compression, syntactic compression and situational compression. The paper mainly discusses the application of compression strategy in E-C discourse under different circumstances.

Latent class model with application to speaker diarization

In this paper, we apply a latent class model (LCM) to the task of speaker diarization. LCM is similar to Patrick Kenny’s variational Bayes (VB) method in that it uses soft information and avoids premature hard decisions in its iterations. In contrast to the VB method, which is based on a generative model, LCM provides a framework allowing both generative and discriminative models. The discriminative property is realized through the use of i-vector (Ivec), probabilistic linear discriminative analysis (PLDA), and a support vector machine (SVM) in this work. Systems denoted as LCM-Ivec-PLDA, LCM-Ivec-SVM, and LCM-Ivec-Hybrid are introduced. In addition, three further improvements are applied to enhance its performance. (1) Adding neighbor windows to extract more speaker information for each short segment. (2) Using a hidden Markov model to avoid frequent speaker change points. (3) Using an agglomerative hierarchical cluster to do initialization and present hard and soft priors, in order to overcome the problem of initial sensitivity. Experiments on the National Institute of Standards and Technology Rich Transcription 2009 speaker diarization database, under the condition of a single distant microphone, show that the diarization error rate (DER) of the proposed methods has substantial relative improvements compared with mainstream systems. Compared to the VB method, the relative improvements of LCM-Ivec-PLDA, LCM-Ivec-SVM, and LCM-Ivec-Hybrid systems are 23.5%, 27.1%, and 43.0%, respectively. Experiments on our collected database, CALLHOME97, CALLHOME00, and SRE08 short2-summed trial conditions also show that the proposed LCM-Ivec-Hybrid system has the best overall performance.

Addressing Text-Dependent Speaker Verification Using Singing Speech

The automatic speaker verification (ASV) has achieved significant progress in recent years. However, it is still very challenging to generalize the ASV technologies to new, unknown and spoofing conditions. Most previous studies focused on extracting the speaker information from natural speech. This paper attempts to address the speaker verification from another perspective. The speaker identity information was exploited from singing speech. We first designed and released a new corpus for speaker verification based on singing and normal reading speech. Then, the speaker discrimination was compared and analyzed between natural and singing speech in different feature spaces. Furthermore, the conventional Gaussian mixture model, the dynamic time warping and the state-of-the-art deep neural network were investigated. They were used to build text-dependent ASV systems with different training-test conditions. Experimental results show that the voiceprint information in the singing speech was more distinguishable than the one in the normal speech. More than relative 20% reduction of equal error rate was obtained on both the gender-dependent and independent 1 s-1 s evaluation tasks.

The Conceptualisation of FEAR through Conceptual Metonymy and Metaphor in Jordanian Arabic

This study aims to examine the figurative devices used in Jordanian Arabic (JA) to conceptualise the emotion of FEAR. It investigates whether FEAR in JA can be conceptualised: (1) universally, on the basis of human embodied experiences; and (2) socioculturally, on the basis of culture specific schemas. The study adopts Conceptual Metaphor Theory (CMT) as its main theoretical framework to compare and contrast the similarities and/or differences in the conceptualisations of FEAR in both JA and English. Data are collected from two sources: namely, the comment section on Facebook pages of cinemas in Amman that show horror and thriller movies, and 12 JA native speaker informants. Data analysis of the linguistic metonymical and metaphorical expressions shows that three figurative devices are used to conceptualise FEAR in JA: Conceptual metonymy, conceptual metaphor and conceptual metaphtonymy. Data analysis of the linguistic realisations in JA and English also demonstrate that both languages exhibit similarities and differences in the conceptualisations of FEAR. The similarities between the two languages are explained by relying on universal embodied cognition, whereas the differences are accounted for in terms of cultural filtering of general universal conceptualisations, reflecting experiences that are more common to a particular sociocultural group..

Articulatory and bottleneck features for speaker-independent ASR of dysarthric speech

The rapid population aging has stimulated the development of assistive devices that provide personalized medical support to the needies suffering from various etiologies. One prominent clinical application is a computer-assisted speech training system which enables personalized speech therapy to patients impaired by communicative disorders in the patient’s home environment. Such a system relies on the robust automatic speech recognition (ASR) technology to be able to provide accurate articulation feedback. With the long-term aim of developing off-the-shelf ASR systems that can be incorporated in clinical context without prior speaker information, we compare the ASR performance of speaker-independent bottleneck and articulatory features on dysarthric speech used in conjunction with dedicated neural network-based acoustic models that have been shown to be robust against spectrotemporal deviations. We report ASR performance of these systems on two dysarthric speech datasets of different characteristics to quantify the achieved performance gains. Despite the remaining performance gap between the dysarthric and normal speech, significant improvements have been reported on both datasets using speaker-independent ASR architectures.