Voice Of Speaker Research Articles

HiddenSinger: High-quality singing voice synthesis via neural audio codec and latent diffusion models

Recently, denoising diffusion models have demonstrated remarkable performance among generative models in various domains. However, in the speech domain, there are limitations in complexity and controllability to apply diffusion models for time-varying audio synthesis. Particularly, a singing voice synthesis (SVS) task, which has begun to emerge as a practical application in the game and entertainment industries, requires high-dimensional samples with long-term acoustic features. To alleviate the challenges posed by model complexity in the SVS task, we propose HiddenSinger, a high-quality SVS system using a neural audio codec and latent diffusion models. To ensure high-fidelity audio, we introduce an audio autoencoder that can encode audio into an audio codec as a compressed representation and reconstruct the high-fidelity audio from the low-dimensional compressed latent vector. Subsequently, we use the latent diffusion models to sample a latent representation from a musical score. In addition, our proposed model is extended to an unsupervised singing voice learning framework, HiddenSinger-U, to train the model using an unlabeled singing voice dataset. Experimental results demonstrate that our model outperforms previous models regarding audio quality. Furthermore, the HiddenSinger-U can synthesize high-quality singing voices of speakers trained solely on unlabeled data.

Bunyi Forman Vokal dalam Bahasa Jawa Ngoko: Kajian Fonetik Akustik

The Javanese language (BJ) is divided into three, namely Ngoko, Krama Madya, and Krama Inggil. BJ consists of 10 vowel sounds: [i, I, e, ɛ, a, ǝ, o, ɔ, U, u]. This study aims to determine the values of BJ vowel sounds based on FORMAN data analysis, comparing the voices of male speakers (PL) and female speakers (PP). Additionally, the study aims to identify the more dominant vowel sounds based on gender. The qualitative method was used to collect data on the vowel sounds of Ngoko Javanese (BJN) by analyzing voice recordings using the Praat application program. The quantitative method was employed to calculate the FORMAN F1, F2, and F3 values and determine the average difference in BJN vowel sounds. Data from the BJ dictionary, consisting of 10 words with vowel sounds at the beginning, were used for analysis. The results showed that in PL, the highest vowel values were: [u] at 3089Hz, [i] at 3918Hz, [ɔ] at 3535Hz, and [ɛ] at 3228Hz. The lowest vowel values were: [ɛ] at 277Hz, [U] at 431Hz, and [i] at 336Hz. In PP, the highest vowel values were: [ɛ] at 3021Hz, [i] at 3200Hz, [I] at 3083Hz, and [o] at 3367Hz. The lowest vowel values were: [e] at 391Hz, [I] at 525Hz, [ɛ] at 385Hz, and [i] at 455Hz. In conclusion, BJN consists of 10 vowel sounds, with the highest vowel sound being [ɔ] and the lowest vowel sound being [i] in PL. The comparison of vowel sounds at the beginning of words in PL and PP in BJN, based on the high and low sounds of the F1, F2, and F3 vowel sounds, indicates that PL is more dominant.

Memorization-Based Training and Testing Paradigm for Robust Vocal Identity Recognition in Expressive Speech Using Event-Related Potentials Analysis.

Recognizing familiar speakers from vocal streams is a fundamental aspect of human verbal communication. However, it remains unclear how listeners can still discern the speaker's identity in expressive speech. This study develops a memorization-based individual speaker identity recognition approach and an accompanying electroencephalogram (EEG) data analysis pipeline, which monitors how listeners recognize familiar speakers and tell unfamiliar ones apart. EEG data captures online cognitive processes during new versus old speaker distinction based on voice, offering a real-time measure of brain activity, overcoming limits of reaction times and accuracy measurements. The paradigm comprises three steps: listeners establish associations between three voices and their names (training); listeners indicate the name corresponding to a voice from three candidates (checking); listeners distinguish between three old and three new speaker voices in a two-alternative forced-choice task (testing). The speech prosody in testing was either confident or doubtful. EEG data were collected using a 64-channel EEG system, followed by preprocessing and imported into RStudio for ERP and statistical analysis and MATLAB for brain topography. Results showed an enlarged late positive component (LPC) was elicited in the old-talker compared to the new-talker condition in the 400-850 ms window in the Pz and other wider range of electrodes in both prosodies. Yet, the old/new effect was robust in central and posterior electrodes for doubtful prosody perception, whereas the anterior, central, and posterior electrodes are for confident prosody condition. This study proposes that this experiment design can serve as a reference for investigating speaker-specific cue-binding effects in various scenarios (e.g., anaphoric expression) and pathologies in patients like phonagnosia.

Lombard speech detection in case of spatial separation between noise source and talkers of different genders

The spatial selectivity of hearing to speech signals was studied when the target signal and interference were separated by distance between their sources and the listener. In the work, the hypothesis about the improvement of hearing selectivity due to more intensive activation of the high-frequency binaural mechanism due to the shift of the speaker’s voice spectrum occurs in noisy environment towards high frequencies, was tested. The thresholds for detecting the target signal – a two-syllable word uttered by male or female, were evaluated in the two-alternative two-interval forced choice paradigm in 4 series. Series differed by the type of target signal (ordinary or Lombard speech) and the location of target source and noise (multi-talker noise) one. The both sources were located at a distance of 1 and 4 m opposite the subject at the level of his head. The detection threshold was defined as the ratio of signal and noise levels at the listener’s place (S/N). The threshold for detecting the target signal (male and female speaker voices together) was -11 dB S/N for ordinary as well as Lombard speech. It did not depend on which of the sources - the target signal or noise, was closer to the listener. In normal speech, the detection thresholds on average differed for male and female voices, but the difference was not significant. In Lombard speech, these thresholds were significantly different: for a male voice, the threshold at a detection level of 0.67 was -10 dB S/N, and for a female voice – -12.5 dB S/N.

Using deep learning to improve the intelligibility of a target speaker in noisy multi-talker environments for people with normal hearing and hearing loss.

Understanding speech in noisy environments is a challenging task, especially in communication situations with several competing speakers. Despite their ongoing improvement, assistive listening devices and speech processing approaches still do not perform well enough in noisy multi-talker environments, as they may fail to restore the intelligibility of a speaker of interest among competing sound sources. In this study, a quasi-causal deep learning algorithm was developed that can extract the voice of a target speaker, as indicated by a short enrollment utterance, from a mixture of multiple concurrent speakers in background noise. Objective evaluation with computational metrics demonstrated that the speaker-informed algorithm successfully extracts the target speaker from noisy multi-talker mixtures. This was achieved using a single algorithm that generalized to unseen speakers, different numbers of speakers and relative speaker levels, and different speech corpora. Double-blind sentence recognition tests on mixtures of one, two, and three speakers in restaurant noise were conducted with listeners with normal hearing and listeners with hearing loss. Results indicated significant intelligibility improvements with the speaker-informed algorithm of 17% and 31% for people without and with hearing loss, respectively. In conclusion, it was demonstrated that deep learning-based speaker extraction can enhance speech intelligibility in noisy multi-talker environments where uninformed speech enhancement methods fail.

Perception of Voice Attractiveness: Effects of Speaking Rate, Gender, and Age

Objectives: The purpose of this study was to investigate the effects of the speaker’s speech rate level, listener’s age, and gender on the listener’s attractiveness evaluation of the speaker’s voice. Methods: A total of 8 speaker voices (4 males and 4 females in their 20s-30s) were selected from the speech corpus provided by the National Institute of Korean Language. These voices were manipulated to be 0.8 times and 1.2 times their original speed, resulting in 24 voice files for attractiveness evaluation. Evaluators were recruited from different age groups: adolescents (14), young adults (16), and middle-aged adults (14). They were instructed to perceptually evaluate the attractiveness of the speaker’s voice. In addition, speech data from evaluators were collected to measure their actual speech rates. Results: Evaluator age groups did not show significant differences in attractiveness scores. However, across all groups, the attractiveness ratings were consistently higher for normal speech rates. Middle-aged male evaluators reported a significant decrease in attractiveness for faster speech rates. The correlation between the actual speech rate of evaluators and speakers was significant only for slower speech rates, indicating that evaluators with faster speech rates, especially males, showed lower attractiveness. Conclusion: The listener’s attractiveness evaluation may depend on the listener’s gender and age according to the speaker’s speaking rate, with middle-aged male listeners being particularly inclined to find fast speech rates unacceptable. This study underscores the significance of speech rate as an acoustic evaluation factor influencing listener attractiveness.

Who is singing? Voice recognition from spoken versus sung speech.

Singing is socially important but constrains voice acoustics, potentially masking certain aspects of vocal identity. Little is known about how well listeners extract talker details from sung speech or identify talkers across the sung and spoken modalities. Here, listeners (n = 149) were trained to recognize sung or spoken voices and then tested on their identification of these voices in both modalities. Learning vocal identities was initially easier through speech than song. At test, cross-modality voice recognition was above chance, but weaker than within-modality recognition. We conclude that talker information is accessible in sung speech, despite acoustic constraints in song.

Tolkien and Voice

In this article at the intersection of literary studies and sound studies, I extend academic analysis of sound in modernist literature to the fantasy genre, focusing on the treatment of sound and voices in relation to the narrative building of invented worlds. Not only is sound an arguably under-analysed dimension of literary text, but the affective potential of reading, the sound of specific words, and the distribution and description of voices have wide-reaching implications for the ways authors can both subtly and overtly influence a reader’s reception of a text, a character, a scene, and – in the case of fantasy literature – entire invented races. I demonstrate how sound can play a unique role in fantasy worldbuilding and the potential narrative impact which sounds – if any – an author chooses to include or highlight in their work. Using the sound studies concept of ‘voice’, I analyse examples from J.R.R. Tolkien’s The Lord of the Rings trilogy to show the way that descriptions of voices signpost whether a character or race is friendly or antagonistic and explore the possible affective influence of such choices on the reader’s judgments. More specifically, I focus on the contrast between Saruman’s ‘enchanting’ voice which leads the listener astray and Gandalf’s ‘clear’ one countering it, as well as the descriptions of sounds emitted by the Black Riders versus the Ents, after a brief examination of which invented races get to speak in the first place or make other characteristic sounds. Thus, I illustrate the interplay of sound with fantasy worldbuilding, the importance of an author’s choice in who to give a speaking voice (or any sound at all), and the potential effect of sound descriptions on the reader.

Speech Cloning: Text-To-Speech Using VITS

Voice is one of the most common and natural communication methods for humans. Voice is becoming the primary interface for AI voice assistants like Amazon Alexa, as well as in autos and smart home devices. Homes and so on. As human-machine communication becomes more common, researchers are exploring technology that mimics genuine speech. Speech cloning is the practice of copying or mimicking another person's speech, usually utilizing modern technology and artificial intelligence (AI). This entails producing a synthetic or cloned version of someone's voice that sounds very similar to the actual speaker. The objective is to produce speech that is indistinguishable from the genuine person, both in tone and intonation. Instant Voice Cloning (IVC) in text-to-speech (TTS) synthesis refers to the TTS model's capacity to copy the voice of any reference speaker based on a short audio sample, without requiring extra speaker-specific training. This method is usually referred to as zero-shot TTS. IVC provides users with the flexibility to tailor the generated voice, offering significant value across diverse real-world applications. Examples include media content creation, personalized chatbots, and multi-modal interactions between humans and computers or extensive language models.

Professional voice use in health and nursing care - Time for reconsideration? A scoping review.

Communication is a key tool in the nursing profession. It is known that listeners are sensitive to the speaker's voice and interpret the speaker's intentions primarily from the non-verbal signal conveyed. To map and discuss the current state of knowledge and research evidence on professional voice use in health and nursing care. A scoping review adherent to the EQUATOR guidelines PRISMA-ScR. Searches in the EBSCO, CINAHL, Medline, PubMed, PsychInfo and PsycArticles databases were conducted between 1 and 28 February 2023. According to the data (n = 23), health professionals use voice as a vital aspect of caring communication and caring relationships, and in leadership as well as inter-professional collaboration, to facilitate information exchange and understanding and promote shared decision-making. The data showed that there is also a rising tendency to explore vocal demands and problems among health professionals. No studies were found on the use of voice in digital health and nursing care services or by social robots. Most of the reviewed studies were cross-sectional, rather small, and focused primarily on the perspectives of healthcare professionals. The use of voice should be explored as a vital aspect of caring communication and caring relationships from the patients' perspective, and the effects of voice use on listeners' emotions and actions should be examined in different contexts with modern, voice-specific data collection methods. Studies focusing on voice use in digital health care and by social robots are also needed. By integrating training programmes, education and technological innovations, health care can leverage the full potential of voice-based communication to promote a more coordinated and patient-centred care environment, true inter-professional collaboration and effective leadership. Professional voice use is an essential part of all health and nursing care and an impressive method that should be used consciously. Therefore, the meaning and methods of voice use, including speech accommodation, should be systematically introduced into health and nursing care and included in nursing education.

MRMI-TTS: Multi-Reference Audios and Mutual Information Driven Zero-Shot Voice Cloning

Voice cloning in text-to-speech (TTS) is the process of replicating the voice of a target speaker with limited data. Among various voice cloning techniques, this article focuses on zero-shot voice cloning. Although existing TTS models can generate high-quality speech for seen speakers, cloning the voice of an unseen speaker remains a challenging task. The key aspect of zero-shot voice cloning is to obtain a speaker embedding from the target speaker. Previous works have used a speaker encoder to obtain a fixed-size speaker embedding from a single reference audio unsupervised, but they suffer from insufficient speaker information and content information leakage in speaker embedding. To address these issues, this article proposes MRMI-TTS, a FastSpeech2-based framework that uses speaker embedding as a conditioning variable to provide speaker information. The MRMI-TTS extracts speaker embedding and content embedding from multi-reference audios using a speaker encoder and a content encoder. To obtain sufficient speaker information, multi-reference audios are selected based on sentence similarity. The proposed model applies mutual information minimization on the two embeddings to remove entangled information within each embedding. Experiments on the public English dataset VCTK show that our method can improve synthesized speech in terms of both similarity and naturalness, even for unseen speakers. Compared to state-of-the-art reference embedding learned methods, our method achieves the best performance on the zero-shot voice cloning task. Furthermore, we demonstrate that the proposed method has a better capability of maintaining the speaker embedding in different languages. Sample outputs are available on the demo page. 1

Acoustic cues to femininity and masculinity in spontaneous speech.

The perceived level of femininity and masculinity is a prominent property by which a speaker's voice is indexed, and a vocal expression incongruent with the speaker's gender identity can greatly contribute to gender dysphoria. Our understanding of the acoustic cues to the levels of masculinity and femininity perceived by listeners in voices is not well developed, and an increased understanding of them would benefit communication of therapy goals and evaluation in gender-affirming voice training. We developed a voice bank with 132 voices with a range of levels of femininity and masculinity expressed in the voice, as rated by 121 listeners in independent, individually randomized perceptual evaluations. Acoustic models were developed from measures identified as markers of femininity or masculinity in the literature using penalized regression and tenfold cross-validation procedures. The 223 most important acoustic cues explained 89% and 87% of the variance in the perceived level of femininity and masculinity in the evaluation set, respectively. The median fo was confirmed to provide the primary cue, but other acoustic properties must be considered in accurate models of femininity and masculinity perception. The developed models are proposed to afford communication and evaluation of gender-affirming voice training goals and improve voice synthesis efforts.

Face and Speech Emotion Recognition System

Abstract: Emotions serve as the cornerstone of human communication, facilitating the expression of one's inner thoughts and feelings to others. Speech Emotion Recognition (SER) represents a pivotal endeavour aimed at deciphering the emotional nuances embedded within a speaker's voice signal. Universal emotions such as neutrality, anger, happiness, and sadness form the basis of this recognition process, allowing for the identification of fundamental emotional states. To achieve this, spectral and prosodic features are leveraged, each offering unique insights into the emotional content of speech. Spectral features, exemplified by the Mel Frequency Cepstral Coefficient (MFCC), provide a detailed analysis of the frequency distribution within speech signals, while prosodic features encompass elements like fundamental frequency, volume, pitch, speech intensity, and glottal parameters, capturing the rhythmic and tonal variations indicative of different emotional states. Through the integration of these features, SER systems can effectively simulate and classify a diverse range of emotional expressions, paving the way for enhanced human-computer interaction and communication technologies

SVCGAN: Speaker Voice Conversion Generative Adversarial Network for Children’s Speech Conversion and Recognition

Automatic speech recognition (ASR) refers to a technological process that entails the conversion of spoken language into written text. However, the acoustic distinctions between children’s speech and adult speech are substantial, rendering the automatic speech recognition system trained on adult speech inadequate for effectively recognizing children’s speech. To overcome this issue, in this study, we propose speaker conversion generative adversarial network (SVCGAN). SVCGAN is a novel non-parallel voice conversion model, which enhances three key areas: log-cosh loss, semantic-similarity loss, and third adversarial loss. Therefore, the incorporation of these losses better protects semantic information for young children during voice conversion process and improves the quality of the converted speech. Additionally, the character error rate (CER) of children’s speech recognition can benefit from children’s speech transformed into adult speech. Experimental results suggest that SVCGAN demonstrates superior performance across multiple dimensions compared to both CycleGAN-VC3 and MaskCycleGAN-VC models. It encompasses training efficiency, semantic information similarity, voice type similarity, sound naturalness and intelligibility, which leads to a reduction in the CER of speech recognition for young children.

Comparative Analysis Using Machine Learning Algorithms to Detect Parkinson’s Disease using Voice Dataset

Abstract: Parkinson's disease is a degenerative disorder that affects the control of movement, and its symptoms include tremors, stiffness, and difficulties in balance and coordination. This disease results from a decrease in dopamine-producing cells in the brain, and although there is currently no cure for Parkinson's, medication and therapeutic interventions can help alleviate the symptoms. Parkinson's disease usually begins to show its effects in middle to late adulthood. A new measure of dysphonia, Pitch Period Entropy (PPE) assesses the variability or randomness of pitch periods in speech signals, providing insights into the diversity of vocal characteristics. Higher entropy values suggest greater pitch variability in a speaker's voice. PPE is included in the dataset. The methodologies used in this project are Random Forest, Support vector classifier, Extreme gradient boosting (XGB). This comparative study includes performance metrics: Accuracy, precision, Recall, f1 score, and support. The aim is to predict Parkinson's disease and compare performance metrics using various methods.

Phoneme Hallucinator: One-Shot Voice Conversion via Set Expansion

Voice conversion (VC) aims at altering a person's voice to make it sound similar to the voice of another person while preserving linguistic content. Existing methods suffer from a dilemma between content intelligibility and speaker similarity; i.e., methods with higher intelligibility usually have a lower speaker similarity, while methods with higher speaker similarity usually require plenty of target speaker voice data to achieve high intelligibility. In this work, we propose a novel method Phoneme Hallucinator that achieves the best of both worlds. Phoneme Hallucinator is a one-shot VC model; it adopts a novel model to hallucinate diversified and high-fidelity target speaker phonemes based just on a short target speaker voice (e.g. 3 seconds). The hallucinated phonemes are then exploited to perform neighbor-based voice conversion. Our model is a text-free, any-to-any VC model that requires no text annotations and supports conversion to any unseen speaker. Quantitative and qualitative evaluations show that Phoneme Hallucinator outperforms existing VC methods for both intelligibility and speaker similarity.

Identity, Gender, Age, and Emotion Recognition from Speaker Voice with Multi-task Deep Networks for Cognitive Robotics

This paper presents a study on the use of multi-task neural networks (MTNs) for voice-based soft biometrics recognition, e.g., gender, age, and emotion, in social robots. MTNs enable efficient analysis of audio signals for various tasks on low-power embedded devices, thus eliminating the need for cloud-based solutions that introduce network latency. However, the strict dataset requirements for training limit the potential of MTNs, which are commonly used to optimize a single reference problem. In this paper, we propose three MTN architectures with varying accuracy-complexity trade-offs for voice-based soft biometrics recognition. In addition, we adopt a learnable voice representation, that allows to adapt the specific cognitive robotics application to the environmental conditions. We evaluate the performance of these models on standard large-scale benchmarks, and our results show that the proposed architectures outperform baseline models for most individual tasks. Furthermore, one of our proposed models achieves state-of-the-art performance on three out of four of the considered benchmarks. The experimental results demonstrate that the proposed MTNs have the potential for being part of effective and efficient voice-based soft biometrics recognition in social robots.

Cross-lingual speaker transfer for Cambodian based on feature disentangler and time-frequency attention adaptive normalization

PurposeThis paper aims to disentangle Chinese-English-rich resources linguistic and speaker timbre features, achieving cross-lingual speaker transfer for Cambodian.Design/methodology/approachThis study introduces a novel approach: the construction of a cross-lingual feature disentangler coupled with the integration of time-frequency attention adaptive normalization to proficiently convert Cambodian speaker timbre into Chinese-English without altering the underlying Cambodian speech content.FindingsConsidering the limited availability of multi-speaker corpora in Cambodia, conventional methods have demonstrated subpar performance in Cambodian speaker voice transfer.Originality/valueThe originality of this study lies in the effectiveness of the disentanglement process and precise control over speaker timbre feature transfer.

Speaker voice normalization for end-to-end speech translation

Speaker voices exhibit acoustic variation. Our preliminary experiments reveal that normalized voice can significantly improve end-to-end speech translation. To mitigate the negative impact of acoustic voice variation across speakers on speech translation, we propose SVN-ST, a Speaker-Voice-Normalized end-to-end Speech Translation framework. In SVN-ST, we use synthetic speech inputs generated from a Text-to-Speech system to complement raw speech inputs. In order to explore synthetic speech inputs, we introduce two essential components for SVN-ST: an alignment adapter at the encoder side and a normalized speech knowledge distillation module at the decoder side. The former forces the representations of raw speech inputs to be close to those of synthetic (normalized) speech inputs while the latter attempts to guide the translations of raw speech inputs with those yielded from synthetic speech inputs. Two additional losses are also defined to equip with the two components. Experimental results on the MuST-C benchmark dataset demonstrate that SVN-ST outperforms previous state-of-the-art end-to-end non-normalized speech translation systems by 0.4 BLEU and cascaded speech translation systems by 2.3 BLEU. On the Covost 2 testset, SVN-ST also outperforms other normalized speech methods on robustness. Further analyses suggest that our model effectively aligns speech representations from different speakers, enhances robustness, and significantly improves sentence-level translation quality.

Formant Analysis in Native and Non-Native Speakers

This research wants to find out how the voice formants of people who speak English as their first language differ from the voices of people who speak English as a second language. This research uses a computer program called Praat to help with the research. The people to be studied are those who grew up speaking English and those who learned it later in life. The samples in this research were taken 1 male voice and 1 female voice non-native speaker and 2 native speaker voice sample Male and Female. The findings of the analysis show that there are not many differences between native and non-native speakers' pronunciations of each monophthong, especially when they relate to the monophthong /ə/, which is difficult for non-native speakers to pronounce correctly.