Speech Processing Tasks Research Articles

Comparison of wav2vec 2.0 models on three speech processing tasks

Comparison of wav2vec 2.0 models on three speech processing tasks

Impaired speech input and output processing abilities in children with cleft palate speech disorder.

Cleft lip and palate is one of the most common oral and maxillofacial deformities associated with a variety of functional disorders. Cleft palate speech disorder (CPSD) occurs the most frequently and manifests a series of characteristic speech features, which are called cleft speech characteristics. Some scholars believe that children with CPSD and poor speech outcomes may also have weaknesses in speech input processing ability, but evidence is still lacking so far. (1) To explore whether children with CPSD and speech output disorders also have defects in speech input processing abilities; (2) to explore the correlation between speech input and output processing abilities. Children in the experimental group were enrolled from Beijing Stomatological Hospital, Capital Medical University, and healthy volunteers were recruited as controls. Then three tasks containing real and pseudo words were performed sequentially. Reaction time, accuracy and other indicators in three tasks were collected and then analysed. The indicators in the experimental group were significantly lower than those in the control group. There was a strong correlation between speech input and output processing tasks. The performance of both groups when processing pseudo words in the three tasks was worse than that when dealing with real words. Compared with normal controls, children with CPSD have deficits in both speech input and output processing, and there is a strong correlation between speech input and output speech processing abilities. In addition, the pseudo words task was more challenging than the real word task for both groups. What is already known on the subject Children with cleft lip and palate often have speech sound disorders known as cleft palate speech disorder (CPSD). CPSD is characterised by consonant errors called cleft speech characteristics, which can persist even after surgery. Some studies suggest that poor speech outcomes in children with CPSD may be associated with deficits in processing speech input. However, this has not been validated in mainland China. What this paper adds to existing knowledge The results of our study indicate that children with CPSD exhibit poorer performance in three tasks assessing speech input and output abilities compared to healthy controls, suggesting their deficits in both speech input and output processing. Furthermore, a significant correlation was observed between speech input and output processing abilities. Additionally, both groups demonstrated greater difficulty in processing pseudo words compared to real words, as evidenced by their worse performance in dealing with pseudo words. What are the potential or actual clinical implications of this work? The pseudo word tasks designed and implemented in our study can be employed in future research and assessment of speech input and output abilities in Chinese Mandarin children with CPSD. Additionally, our findings revealed the significance of considering both speech output processing abilities and potential existence of speech input processing ability for speech and language therapists when evaluating and developing treatment options for children with CPSD as these abilities are also important for the development of literacy development.

Effective Monoaural Speech Separation through Convolutional Top-Down Multi-View Network

Speech separation, sometimes known as the “cocktail party problem”, is the process of separating individual speech signals from an audio mixture that includes ambient noises and several speakers. The goal is to extract the target speech in this complicated sound scenario and either make it easier to understand or increase its quality so that it may be used in subsequent processing. Speech separation on overlapping audio data is important for many speech-processing tasks, including natural language processing, automatic speech recognition, and intelligent personal assistants. New speech separation algorithms are often built on a deep neural network (DNN) structure, which seeks to learn the complex relationship between the speech mixture and any specific speech source of interest. DNN-based speech separation algorithms outperform conventional statistics-based methods, although they typically need a lot of processing and/or a larger model size. This study presents a new end-to-end speech separation network called ESC-MASD-Net (effective speaker separation through convolutional multi-view attention and SuDoRM-RF network), which has relatively fewer model parameters compared with the state-of-the-art speech separation architectures. The network is partly inspired by the SuDoRM-RF++ network, which uses multiple time-resolution features with downsampling and resampling for effective speech separation. ESC-MASD-Net incorporates the multi-view attention and residual conformer modules into SuDoRM-RF++. Additionally, the U-Convolutional block in ESC-MASD-Net is refined with a conformer layer. Experiments conducted on the WHAM! dataset show that ESC-MASD-Net outperforms SuDoRM-RF++ significantly in the SI-SDRi metric. Furthermore, the use of the conformer layer has also improved the performance of ESC-MASD-Net.

Gammatonegram representation for end-to-end dysarthric speech processing tasks: speech recognition, speaker identification, and intelligibility assessment

Gammatonegram representation for end-to-end dysarthric speech processing tasks: speech recognition, speaker identification, and intelligibility assessment

A Bilingual Basque–Spanish Dataset of Parliamentary Sessions for the Development and Evaluation of Speech Technology

The development of speech technology requires large amounts of data to estimate the underlying models. Even when relying on large multilingual pre-trained models, some amount of task-specific data on the target language is needed to fine-tune those models and obtain competitive performance. In this paper, we present a bilingual Basque–Spanish dataset extracted from parliamentary sessions. The dataset is designed to develop and evaluate automatic speech recognition (ASR) systems but can be easily repurposed for other speech-processing tasks (such as speaker or language recognition). The paper first compares the two target languages, emphasizing their similarities at the acoustic-phonetic level, which sets the basis for sharing data and compensating for the relatively small amount of spoken resources available for Basque. Then, Basque Parliament plenary sessions are characterized in terms of organization, topics, speaker turns and the use of the two languages. The paper continues with the description of the data collection procedure (involving both speech and text), the audio formats and conversions along with the creation and postprocessing of text transcriptions and session minutes. Then, it describes the semi-supervised iterative procedure used to cut, rank and select the training segments and the manual supervision process employed to produce the test set. Finally, ASR experiments are presented using state-of-the-art technology to validate the dataset and to set a reference for future works. The datasets, along with models and recipes to reproduce the experiments reported in the paper, are released through Hugging Face.

Transfer learning methods for low-resource speech accent recognition: A case study on Vietnamese language

Speech accent recognition (SAR) plays a crucial role in enhancing communication between customers and service providers, enabling personalized interactions based on geographical, birthplace, and cultural cues derived from accents. However, current approaches predominantly rely on training SAR models from scratch, overlooking the potential of transfer learning from other speech processing tasks, despite the relatively small size of accent datasets. This paper represents the first comprehensive investigation into the effectiveness of transfer learning methods derived from a diverse array of data-rich speech processing tasks for SAR. Through experiments on a practical Vietnamese telephone dataset provided by Viettel, the largest telecommunications provider in Southeast Asia, our study reveals that our best-performing model outperforms previous state-of-the-art SAR models by 46.7% in terms of relative accuracy.

Voice separation and recognition using machine learning and deep learning a review paper

Voice isolation, a prominent research area in the field of speech processing, has garnered a great deal of attention due to its prospective implications in numerous domains. Deep neural networks (DNNs) have emerged as a potent instrument for addressing the challenges associated with vocal isolation. This paper presents a comprehensive study on the use of DNNs for voice isolation, focusing on speech recognition and speaker identification tasks. The proposed method uses frequency domain and time domain techniques to improve the separation of target utterances from background noise. The experimental results demonstrate the efficacy of the proposed method, revealing substantial improvements in voice isolation precision and robustness. This study's findings contribute to the increasing corpus of research on voice isolation techniques and provide valuable insights into the application of DNNs to improve speech processing tasks.

Optimizing Voice Recognition Informatic Robots for Effective Communication in Outpatient Settings.

Aim/Objective Within the dynamic healthcare technology landscape, this research aims to explore patient inquiries within outpatient clinics, elucidating the interplay between technology and healthcare intricacies. Building upon the initial intelligent guidance robot implementation shortcomings, this investigation seeks to enhance informatic robots with voice recognition technology. The objective is to analyze users' vocal patterns, discern age-associated vocal attributes, and facilitate age differentiation through subtle vocal nuances to enhance the efficacy of human-robot communication within outpatient clinical settings. Methods This investigation employs a multi-faceted approach. It leverages voice recognition technology to analyze users' vocal patterns. A diverse dataset of voice samples from various age groups was collected. Acoustic features encompassing pitch, formant frequencies, spectral characteristics, and vocal tract length are extracted from the audio samples. The Mel Filterbank and Mel-Frequency Cepstral Coefficients (MFCCs) are employed for speech and audio processing tasks alongside machine learning algorithms to assess and match vocal patterns to age-related traits. Results The research reveals compelling outcomes. The incorporation of voice recognition technology contributes to a significant improvement in human-robot communication within outpatient clinical settings. Through accurate analysis of vocal patterns and age-related traits, informatic robots can differentiate age through nuanced verbal cues. This augmentation leads to enhanced contextual understanding and tailored responses, significantly advancing the efficiency of patient interactions with the robots. Conclusion Integrating voice recognition technology into informatic robots presents a noteworthy advancement in outpatient clinic settings. By enabling age differentiation through vocal nuances, this augmentation enhances the precision and relevance of responses. The study contributes to the ongoing discourse on the dynamic evolution of healthcare technology, underscoring the complex synergy between technological progression and the intricate realities within healthcare infrastructure. As healthcare continues to metamorphose, the seamless integration of voice recognition technology marks a pivotal stride in optimizing human-robot communication and elevating patient care within outpatient settings.

A Novel Bi-Dual Inference Approach for Detecting Six-Element Emotions

In recent years, there has been rapid development in machine learning for solving artificial intelligence tasks in various fields, including translation, speech, and image processing. These AI tasks are often interconnected rather than independent. One specific type of relationship is known as structural duality, which exists between multiple pairs of artificial intelligence tasks. The concept of dual learning has gained significant attention in the fields of machine learning, computer vision, and natural language processing. Dual learning involves using primitive tasks (mapping from domains X to Y) and dual tasks (mapping from domains Y to X) to enhance the performance of both tasks. In this study, we propose a general framework called Bi-Dual Inference by combining the principles of dual inference and dual learning. Our framework generates multiple dual models and a primal model by utilizing two dual tasks: sentiment analysis of input text and sentence generation of sentiment labels. We create these model pairs (primal model f, dual model g) by employing different initialization seeds and data access sequences. Each primal and dual model is considered as a distinct LSTM model. By reasoning about a single task with multiple similar models in the same direction, our framework achieves improved classification results. To validate the effectiveness of our proposed model, we conduct experiments on two datasets, namely NLPCC2013 and NLPCC2014. The results demonstrate that our model outperforms the optimal baseline model in terms of the F1 score, achieving an improvement of approximately 5%. Additionally, we provide parameter values for our proposed model, including model iteration analysis, α parameter analysis, λ parameter analysis, batch size analysis, training sentence length analysis, and hidden layer size setting. These experimental results further confirm the effectiveness of our proposed model.

The NEF-SPA Approach as a Framework for Developing a Neurobiologically Inspired Spiking Neural Network Model for Speech Production.

The computer-based simulation of the whole processing route for speech production and speech perception in a neurobiologically inspired way remains a challenge. Only a few neural based models of speech production exist, and these models either concentrate on the cognitive-linguistic component or the lower-level sensorimotor component of speech production and speech perception. Moreover, these existing models are second-generation neural network models using rate-based neuron approaches. The aim of this paper is to describe recent work developing a third-generation spiking-neuron neural network capable of modeling the whole process of speech production, including cognitive and sensorimotor components. Our neural model of speech production was developed within the Neural Engineering Framework (NEF), incorporating the concept of Semantic Pointer Architecture (SPA), which allows the construction of large-scale neural models of the functioning brain based on only a few essential and neurobiologically well-grounded modeling or construction elements (i.e., single spiking neuron elements, neural connections, neuron ensembles, state buffers, associative memories, modules for binding and unbinding of states, modules for time scale generation (oscillators) and ramp signal generation (integrators), modules for input signal processing, modules for action selection, etc.). We demonstrated that this modeling approach is capable of constructing a fully functional model of speech production based on these modeling elements (i.e., biologically motivated spiking neuron micro-circuits or micro-networks). The model is capable of (i) modeling the whole processing chain of speech production and, in part, for speech perception based on leaky-integrate-and-fire spiking neurons and (ii) simulating (macroscopic) speaking behavior in a realistic way, by using neurobiologically plausible (microscopic) neural construction elements. The model presented here is a promising approach for describing speech processing in a bottom-up manner based on a set of micro-circuit neural network elements for generating a large-scale neural network. In addition, the model conforms to a top-down design, as it is available in a condensed form in box-and-arrow models based on functional imaging and electrophysiological data recruited from speech processing tasks.

Stable eye versus mouth preference in a live speech-processing task

Looking at the mouth region is thought to be a useful strategy for speech-perception tasks. The tendency to look at the eyes versus the mouth of another person during speech processing has thus far mainly been studied using screen-based paradigms. In this study, we estimated the eye-mouth-index (EMI) of 38 adult participants in a live setting. Participants were seated across the table from an experimenter, who read sentences out loud for the participant to remember in both a familiar (English) and unfamiliar (Finnish) language. No statistically significant difference in the EMI between the familiar and the unfamiliar languages was observed. Total relative looking time at the mouth also did not predict the number of correctly identified sentences. Instead, we found that the EMI was higher during an instruction phase than during the speech-processing task. Moreover, we observed high intra-individual correlations in the EMI across the languages and different phases of the experiment. We conclude that there are stable individual differences in looking at the eyes versus the mouth of another person. Furthermore, this behavior appears to be flexible and dependent on the requirements of the situation (speech processing or not).

Random Cycle Loss and Its Application to Voice Conversion

Speech disentanglement aims to decompose independent causal factors of speech signals into separate codes. Perfect disentanglement benefits to a broad range of speech processing tasks. This paper presents a simple but effective disentanglement approach based on cycle consistency loss and random factor substitution. This leads to a novel random cycle (RC) loss that enforces analysis-and-resynthesis consistency, a main principle of reductionism. We theoretically demonstrate that the proposed RC loss can achieve independent codes if well optimized, which in turn leads to superior disentanglement when combined with information bottleneck (IB). Extensive simulation experiments were conducted to understand the properties of the RC loss, and experimental results on voice conversion further demonstrate the practical merit of the proposal. Source code and audio samples can be found on the webpage http://rc.cslt.org.

Does Assessor Masking Affect Kindergartners' Performance on Oral Language Measures? A COVID-19 Era Experiment With Children From Diverse Home Language Backgrounds.

The ongoing COVID-19 pandemic has prompted changes to child assessment procedures in schools such as the use of face masks by assessors. Research with adults suggests that face masks diminish performance on speech processing and comprehension tasks, yet little is known about how assessor masking affects child performance. Therefore, we asked whether assessor masking impacts children's performance on a widely used, individually administered oral language assessment and if impacts vary by child home language background. A total of 96 kindergartners (5-7 years old, n = 45 with a home language other than English) were administered items from the Clinical Evaluation of Language Fundamentals Preschool-Second Edition Recalling Sentences subtest under two conditions: with and without the assessor wearing a face mask. Regression analysis was used to determine if children scored significantly lower in the masked condition and if the effect of masking depended on home language background. Contrary to expectations, we found no evidence that students scored systematically differently in the masked condition. Children with a home language other than English scored lower overall, but masking did not increase the gap in scores by language background. Our results suggest that children's performance on oral language measures is not adversely affected by assessor masking and imply that valid measurements of students' language skills may be obtained in masked conditions. While masking might decrease some of the social determinants of communication (e.g., recognition of emotions), masking in this experiment did not appear to detract from children's ability to hear and immediately recall verbal information. https://doi.org/10.23641/asha.23567463.

A review of deep learning techniques for speech processing

The field of speech processing has undergone a transformative shift with the advent of deep learning. The use of multiple processing layers has enabled the creation of models capable of extracting intricate features from speech data. This development has paved the way for unparalleled advancements in speech recognition, text-to-speech synthesis, automatic speech recognition, and emotion recognition, propelling the performance of these tasks to unprecedented heights. The power of deep learning techniques has opened up new avenues for research and innovation in the field of speech processing, with far-reaching implications for a range of industries and applications. This review paper provides a comprehensive overview of the key deep learning models and their applications in speech-processing tasks. We begin by tracing the evolution of speech processing research, from early approaches, such as MFCC and HMM, to more recent advances in deep learning architectures, such as CNNs, RNNs, transformers, conformers, and diffusion models. We categorize the approaches and compare their strengths and weaknesses for solving speech-processing tasks. Furthermore, we extensively cover various speech-processing tasks, datasets, and benchmarks used in the literature and describe how different deep-learning networks have been utilized to tackle these tasks. Additionally, we discuss the challenges and future directions of deep learning in speech processing, including the need for more parameter-efficient, interpretable models and the potential of deep learning for multimodal speech processing. By examining the field’s evolution, comparing and contrasting different approaches, and highlighting future directions and challenges, we hope to inspire further research in this exciting and rapidly advancing field.

Multi-Scale Feature Learning for Language Identification of Overlapped Speech

Language identification is the front end of multilingual speech-processing tasks. The study aims to enhance the accuracy of language identification in complex acoustic environments by proposing a multi-scale feature extraction method. This method replaces the baseline feature extraction network with a multi-scale feature extraction network (SE-Res2Net-CBAM-BILSTM) to extract multi-scale features. A multilingual cocktail party dataset was simulated, and comparative experiments were conducted with various models. The experimental results show that the proposed model achieved language identification accuracies of 97.6% for an Oriental language dataset and 75% for a multilingual cocktail party dataset Furthermore, comparative experiments show that our model outperformed three other models in the accuracy, recall, and F1 values. Finally, a comparison of different loss functions shows that the model performance was better when using focal loss.

Speaker Verification Based on Single Channel Speech Separation

In multi-speaker scenarios, speech processing tasks like speaker identification and speech recognition are susceptible to noise and overlapped voices. As the overlapped voices are a complicated mixture of signals, a target extraction method from this mixture is a good front end solution for further processing like understanding and classifying. The quality of speech separation can be assessed by the noise ratio or subjective scoring and can also be assessed by accuracy of the downstream processing tasks like speaker identification. In order to make the separation model and speaker identification model more adapted to complex multi-speaker speech overlapping scenarios, this research investigates the speech separation model and incorporate with a voiceprint recognition task. This paper proposes a feature-scale single channel speech separation network connected to a back end speaker verification network with MFCCT feature, so the accuracy of speaker identification indicates the quality of speech separation task. The datasets are prepared by synthesizing Voxceleb1 data, and used for training and testing. The results show that using an objective downstream evaluation can effectively improve the overall performance, as the optimized speech separation model significantly reduced the error rate of speaker verification.

Gammatonegram Representation for End-to-End Dysarthric Speech Processing Tasks: Speech Recognition, Speaker Identification, and Intelligibility Assessment

Gammatonegram Representation for End-to-End Dysarthric Speech Processing Tasks: Speech Recognition, Speaker Identification, and Intelligibility Assessment

Global–Local Self-Attention Based Transformer for Speaker Verification

Transformer models are now widely used for speech processing tasks due to their powerful sequence modeling capabilities. Previous work determined an efficient way to model speaker embeddings using the Transformer model by combining transformers with convolutional networks. However, traditional global self-attention mechanisms lack the ability to capture local information. To alleviate these problems, we proposed a novel global–local self-attention mechanism. Instead of using local or global multi-head attention alone, this method performs local and global attention in parallel in two parallel groups to enhance local modeling and reduce computational cost. To better handle local location information, we introduced locally enhanced location encoding in the speaker verification task. The experimental results of the VoxCeleb1 test set and the VoxCeleb2 dev set demonstrated the improved effect of our proposed global–local self-attention mechanism. Compared with the Transformer-based Robust Embedding Extractor Baseline System, the proposed speaker Transformer network exhibited better performance in the speaker verification task.

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60k hours to 94k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks. The code and pre-trained models are available at https://aka.ms/wavlm.

Speech processing performance of Hungarian-speaking twins and singletons

ABSTRACT Studying speech processing in twins versus their singleton peers provides opportunities to study both genetic and environmental effects on how children acquire these aspects of their speech and – by extension – their phonological systems. Our study focused on speech processing in typically developing Hungarian-speaking twins and their singleton peers between 5 and 9 years of age. Participants included 384 monolingual Hungarian-speaking children (192 twins, and 192 singletons). Data from four tasks – repetition of synthesised monosyllables, nonsense words, well-formed noisy sentences, and well-formed phonologically complex sentences – were analysed. There was a main effect for birth status, and singletons outperformed their twin peers on the majority of the speech processing tasks. Age and task also had effects on the performance of the participants, and there was a three-way task by age by twin versus singleton status indicating that the speech processing performance of twins versus singletons is interdependent with the type of task and age. Our results also indicate that monolingual Hungarian-speaking twins may be at higher risk for developmental speech delays relative to their singleton peers.