Real-world Speech Research Articles

Transfer learning for sparse variable selection in high-dimensional regression from quadratic measurement

In signal processing, the functional relationship between the unknown parameters and response variables extends to the nonlinear forms. A representative model is the quadratic measurement regression. In high-dimensional scenarios, model complexity poses challenges for estimation and variable selection. In this study, we propose a generic transfer learning algorithm and an automatic transferable source detection algorithm to identify helpful sources for the quadratic measurement regression model. The core idea of the general algorithm is based on pre-training and fine-tuning techniques. To our knowledge, this is the first study that introduces transfer learning into this model. Numerous simulation experiments demonstrated the effectiveness of the proposed transfer learning algorithm. In simulation experiments, the proposed transfer algorithm reduced the square-sum error of the results to approximately 1. On a real-world speech dataset, the proposed algorithm significantly improved the accuracy of the reconstructed signals. In the best-case scenario, the minimum mean square error was reduced to 1/4th of that without transfer. The code is available at https://github.com/Sarkovqx/Transfer-QMR.

Real-time Speech Emotion Recognition using Machine Learning

This project is centred around examining spoken language emotion, a critical aspect of human-computer interaction and artificial intelligence. Leveraging advanced techniques such as Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) models, we aim to decipher and understand the nuanced expressions of emotions conveyed through speech. The investigation draws from diverse datasets, including RAVDESS, SAVEE, TESS, and CREAM-D, each carefully chosen to encapsulate a broad spectrum of emotions and real-world speech scenarios. The project unfolds by elucidating the methodologies employed in dataset selection, the prepossessing of raw audio data, and the intricacies of utilizing CNN and LSTM techniques for speech emotion analysis. The main objective is to create a robust and reliable outcome model capable of accurately classifying and interpreting emotions in speech across various contexts. Practical applications of this Research extend to fields such as sentiment analysis and potential contributions to mental health monitoring. Through this project, we contribute valuable insights to the evolving landscape of speech emotion analysis, addressing the inherent challenges and exploring opportunities for enhancing human-computer interaction and emotional intelligence in artificial systems.

Beyond visual primes: Audiovisual integration of racial identity and sentence intelligibility

There is no biological basis for racial categories. Yet, the presentation of racialized stimuli impacts psycholinguistic processing, suggesting that a talker’s racial identity affects measures of intelligibility (Babel & Russell, 2015; McGowan, 2015). Previous work has used static visual primes, presenting images accompanied by acoustic sentence stimuli. Static visual primes may prime expectations of the person’s speech, with unexpected pairings decreasing intelligibility. These findings show that social expectations affect speech intelligibility and have broad implications for real-world speech perception in high-stakes educational (Evans, Munson, & Edwards, 2017), social (Purnell, Baugh, & Idsardi, 1999), and legal (Rickford & King, 2016) contexts. Expanding our understanding of the psycholinguistic and psychophysical mechanisms behind these phenomena requires experimentation with dynamic, audiovisual speech rather than static visuals. The current experiment presents conflicting visual and auditory cues indexing the white and Black racial identities of the talkers. The talker who is seen (the ‘visual’ talker) and the racial identity of the talker who is heard (the ‘audio’ talker) are fully crossed. This design allows us to examine how intelligibility is influenced by visual racial identity, audio racial identity, and the audiovisual (mis)match between them. Data collection is currently in progress and includes groups of both white and Black listeners.

Fidgety Speech Emotion Recognition for Learning Process Modeling

In this paper, the recognition of fidgety speech emotion is studied, and real-world speech emotions are collected to enhance emotion recognition in practical scenarios, especially for cognitive tasks. We first focused on eliciting fidgety emotions and data acquisition for general math learning. Students practice mathematics by performing operations, solving problems, and orally responding to questions, all of which are recorded as audio data. Subsequently, the teacher evaluates the accuracy of these mathematical exercises by scoring, which reflects the cognitive outcomes of the students. Secondly, we propose an end-to-end speech emotion model based on a multi-scale one-dimensional (1-D) residual convolutional neural network. Finally, we conducted an experiment to recognize fidgety speech emotions by testing various classifiers, including SVM, LSTM, 1-D CNN, and the proposed multi-scale 1-D CNN. The experimental results show that the classifier we constructed can identify fidgety emotion well. After conducting a thorough analysis of fidgety emotions and their influence on the learning process, a clear relationship between the two was apparent. The automatic recognition of fidgety emotions is valuable for assisting on-line math teaching.

Recommendations for promoting user agency in the design of speech neuroprostheses.

Brain-computer interfaces (BCI) that directly decode speech from brain activity aim to restore communication in people with paralysis who cannot speak. Despite recent advances, neural inference of speech remains imperfect, limiting the ability for speech BCIs to enable experiences such as fluent conversation that promote agency - that is, the ability for users to author and transmit messages enacting their intentions. Here, we make recommendations for promoting agency based on existing and emerging strategies in neural engineering. The focus is on achieving fast, accurate, and reliable performance while ensuring volitional control over when a decoder is engaged, what exactly is decoded, and how messages are expressed. Additionally, alongside neuroscientific progress within controlled experimental settings, we argue that a parallel line of research must consider how to translate experimental successes into real-world environments. While such research will ultimately require input from prospective users, here we identify and describe design choices inspired by human-factors work conducted in existing fields of assistive technology, which address practical issues likely to emerge in future real-world speech BCI applications.

Semantic processing during continuous speech production: an analysis from eye movements and EEG.

Speech production involves neurological planning and articulatory execution. How speakers prepare for articulation is a significant aspect of speech production research. Previous studies have focused on isolated words or short phrases to explore speech planning mechanisms linked to articulatory behaviors, including investigating the eye-voice span (EVS) during text reading. However, these experimental paradigms lack real-world speech process replication. Additionally, our understanding of the neurological dimension of speech planning remains limited. This study examines speech planning mechanisms during continuous speech production by analyzing behavioral (eye movement and speech) and neurophysiological (EEG) data within a continuous speech production task. The study specifically investigates the influence of semantic consistency on speech planning and the occurrence of "look ahead" behavior. The outcomes reveal the pivotal role of semantic coherence in facilitating fluent speech production. Speakers access lexical representations and phonological information before initiating speech, emphasizing the significance of semantic processing in speech planning. Behaviorally, the EVS decreases progressively during continuous reading of regular sentences, with a slight increase for non-regular sentences. Moreover, eye movement pattern analysis identifies two distinct speech production modes, highlighting the importance of semantic comprehension and prediction in higher-level lexical processing. Neurologically, the dual pathway model of speech production is supported, indicating a dorsal information flow and frontal lobe involvement. The brain network linked to semantic understanding exhibits a negative correlation with semantic coherence, with significant activation during semantic incoherence and suppression in regular sentences. The study's findings enhance comprehension of speech planning mechanisms and offer insights into the role of semantic coherence in continuous speech production. Furthermore, the research methodology establishes a valuable framework for future investigations in this domain.

A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech

Recent Text-to-Speech (TTS) systems trained on reading or acted corpora have achieved near human-level naturalness. The diversity of human speech, however, often goes beyond the coverage of these corpora. We believe the ability to handle such diversity is crucial for AI systems to achieve human-level communication. Our work explores the use of more abundant real-world data for building speech synthesizers. We train TTS systems using real-world speech from YouTube and podcasts. We observe the mismatch between training and inference alignments in mel-spectrogram based autoregressive models, leading to unintelligible synthesis, and demonstrate that learned discrete codes within multiple code groups effectively resolves this issue. We introduce our MQTTS system whose architecture is designed for multiple code generation and monotonic alignment, along with the use of a clean silence prompt to improve synthesis quality. We conduct ablation analyses to identify the efficacy of our methods. We show that MQTTS outperforms existing TTS systems in several objective and subjective measures.

A late fusion deep neural network for robust speaker identification using raw waveforms and gammatone cepstral coefficients

Speaker identification aims at determining the speaker identity by analyzing his voice characteristics, and relies typically on statistical models or machine learning techniques. Frequency-domain features are by far the most used choice to encode the audio input in sound recognition. Recently, some studies have also analyzed the use of time-domain raw waveform (RW) with deep neural network (DNN) architectures. In this paper, we hypothesize that both time-domain and frequency-domain features can be used to increase the robustness of speaker identification task in adverse noisy and reverberation conditions, and we present a method based on a late fusion DNN using RWs and gammatone cepstral coefficients (GTCCs). We analyze the characteristics of RW and spectrum-based short-time features, reporting advantages and limitations, and we show that the joint use can increase the identification accuracy. The proposed late fusion DNN model consists of two independent DNN branches made primarily by convolutional neural networks (CNN) and fully connected neural networks (NN) layers. The two DNN branches have as input short-time RW audio fragments and GTCCs, respectively. The late fusion is computed on the predicted scores of the DNN branches. Since the method is based on short segments, it has the advantage of being independent from the size of the input audio signal, and the identification task can be computed by summing the predicted scores over several short-time frames. Analysis of speaker identification performance computed with simulations show that the late fusion DNN model improves the accuracy rate in adverse noise and reverberation conditions in comparison to the RW, the GTCC, and the mel-frequency cepstral coefficients (MFCCs) features. Experiments with real-world speech datasets confirm the efficiency of the proposed method, especially with small-size audio samples.

The effects of background noise and reverberation on speech perception in cochlear implant users

Indoor noise and reverberation interfere with speech understanding for listeners. Degradation in cochlear implant (CI) processing presents additional challenges for CI users to understand noisy and reverberant speech. Inthis study, we investigated the effect of adverse acoustics on speech perception on three tasks that changed in difficulty and increasing need for top-down processing: (1) word recall, (2) sentence recognition, and (3) conversation comprehension. Each task was tested in five conditions that varied in combinations of adverse acoustics from reverberation (i.e., anechoic, 0.4s RT, vs 1.2s RT) and noise (i.e., quiet versus Speech-shaped-noise at −3dB). We compared performances from normal-hearing (NH) listeners and bilateral CI users between 15 and 25 years old. Preliminary results show that while NH listeners may have some tolerance for speech perception under adverse acoustics, CI users experienced more graded performance deterioration with increasing reverberation and noise. Moreover, we observed task-dependent performance change across adverse acoustic conditions in both NH and CI listeners: While some context (i.e., sentences) may help speech perception, comprehension based on pragmatic language processing (i.e., conversations) is vulnerable under difficult listening situations. These results provide insights on how NH and CI listeners differ in their access to top-down processing in real-world speech perception.

Differentially Private Speaker Anonymization

Sharing real-world speech utterances is key to the training and deployment of voice-based services. However, it also raises privacy risks as speech contains a wealth of personal data. Speaker anonymization aims to remove speaker information from a speech utterance while leaving its linguistic and prosodic attributes intact. State-of-the-art techniques operate by disentangling the speaker information (represented via a speaker embedding) from these attributes and re-synthesizing speech based on the speaker embedding of another speaker. Prior research in the privacy community has shown that anonymization often provides brittle privacy protection, even less so any provable guarantee. In this work, we show that disentanglement is indeed not perfect: linguistic and prosodic attributes still contain speaker information. We remove speaker information from these attributes by introducing differentially private feature extractors based on an autoencoder and an automatic speech recognizer, respectively, trained using noise layers. We plug these extractors in the state-of-the-art anonymization pipeline and generate, for the first time, private speech utterances with a provable upper bound on the speaker information they contain. We evaluate empirically the privacy and utility resulting from our differentially private speaker anonymization approach on the LibriSpeech data set. Experimental results show that the generated utterances retain very high utility for automatic speech recognition training and inference, while being much better protected against strong adversaries who leverage the full knowledge of the anonymization process to try to infer the speaker identity.

Domain Adaptation with Augmented Data by Deep Neural Network Based Method Using Re-Recorded Speech for Automatic Speech Recognition in Real Environment

The most effective automatic speech recognition (ASR) approaches are based on artificial neural networks (ANN). ANNs need to be trained with an adequate amount of matched conditioned data. Therefore, performing training adaptation of an ASR model using augmented data of matched condition as the real environment gives better results for real data. Real-world speech recordings can vary in different acoustic aspects depending on the recording channels and environment such as the Long Term Evolution (LTE) channel of mobile telephones, where data are transmitted with voice over LTE (VoLTE) technology, wireless pin mics in a classroom condition, etc. Acquiring data with such variation is costly. Therefore, we propose training ASR models with simulated augmented data and fine-tune them for domain adaptation using deep neural network (DNN)-based simulated data along with re-recorded data. DNN-based feature transformation creates realistic speech features from recordings of clean conditions. In this research, a comparative investigation is performed for different recording channel adaptation methods for real-world speech recognition. The proposed method yields 27.0% character error rate reduction (CERR) for the DNN-hidden Markov model (DNN-HMM) hybrid ASR approach and 36.4% CERR for the end-to-end ASR approach for the target domain of the LTE channel of telephone speech.

The impact of age on the treatment of late-acquired sounds in children with speech sound disorders.

The question of 'when' to treat speech sounds is often posed in the context of normative data. The new normative data suggest that speech sounds such as /ɹ/ and /l/ are acquired earlier than previously thought. The present study compared the treatment of late-acquired sounds between two age groups of English-speaking children: Young children (4-5) and Old children (7-8). Eight monolingual children with speech sound disorder (SSD) participated in the study. Each child received a criterion-based, standardised, two-phase therapy protocol. Treatment efficacy was measured by examining children's accuracy on real world speech probes. Treatment efficiency was measured by calculating the number of sessions required to meet the exit criterion and the mean session duration. For treatment efficacy, young children learned treated sounds as effectively as oldchildren did. For treatment efficiency, both groups required a comparable number of sessions, but young children required longer sessions than old children. The results suggest that delaying treatment of individual speech sounds is unnecessary and that a range of sounds should be considered as potential treatment targets.

Parametric Cognitive Load Reveals Hidden Costs in the Neural Processing of Perfectly Intelligible Degraded Speech.

Speech is often degraded by environmental noise or hearing impairment. People can compensate for degradation, but this requires cognitive effort. Previous research has identified frontotemporal networks involved in effortful perception, but materials in these works were also less intelligible, and so it is not clear whether activity reflected effort or intelligibility differences. We used functional magnetic resonance imaging to assess the degree to which spoken sentences were processed under distraction and whether this depended on speech quality even when intelligibility of degraded speech was matched to that of clear speech (close to 100%). On each trial, male and female human participants either attended to a sentence or to a concurrent multiple object tracking (MOT) task that imposed parametric cognitive load. Activity in bilateral anterior insula reflected task demands; during the MOT task, activity increased as cognitive load increased, and during speech listening, activity increased as speech became more degraded. In marked contrast, activity in bilateral anterior temporal cortex was speech selective and gated by attention when speech was degraded. In this region, performance of the MOT task with a trivial load blocked processing of degraded speech, whereas processing of clear speech was unaffected. As load increased, responses to clear speech in these areas declined, consistent with reduced capacity to process it. This result dissociates cognitive control from speech processing; substantially less cognitive control is required to process clear speech than is required to understand even very mildly degraded, 100% intelligible speech. Perceptual and control systems clearly interact dynamically during real-world speech comprehension.SIGNIFICANCE STATEMENT Speech is often perfectly intelligible even when degraded, for example, by background sound, phone transmission, or hearing loss. How does degradation alter cognitive demands? Here, we use fMRI to demonstrate a novel and critical role for cognitive control in the processing of mildly degraded but perfectly intelligible speech. We compare speech that is matched for intelligibility but differs in putative control demands, dissociating cognitive control from speech processing. We also impose a parametric cognitive load during perception, dissociating processes that depend on tasks from those that depend on available capacity. Our findings distinguish between frontal and temporal contributions to speech perception and reveal a hidden cost to processing mildly degraded speech, underscoring the importance of cognitive control for everyday speech comprehension.

Measuring Speech Intelligibility and Hearing-Aid Benefit Using Everyday Conversational Sentences in Real-World Environments.

Laboratory and clinical-based assessments of speech intelligibility must evolve to better predict real-world speech intelligibility. One way of approaching this goal is to develop speech intelligibility tasks that are more representative of everyday speech communication outside the laboratory. Here, we evaluate speech intelligibility using both a standard sentence recall task based on clear, read speech (BKB sentences), and a sentence recall task consisting of spontaneously produced speech excised from conversations which took place in realistic background noises (ECO-SiN sentences). The sentences were embedded at natural speaking levels in six realistic background noises that differed in their overall level, which resulted in a range of fixed signal-to-noise ratios. Ten young, normal hearing participants took part in the study, along with 20 older participants with a range of levels of hearing loss who were tested with and without hearing-aid amplification. We found that scores were driven by hearing loss and the characteristics of the background noise, as expected, but also strongly by the speech materials. Scores obtained with the more realistic sentences were generally lower than those obtained with the standard sentences, which reduced ceiling effects for the majority of environments/listeners (but introduced floor effects in some cases). Because ceiling and floor effects limit the potential for observing changes in performance, benefits of amplification were highly dependent on the speech materials for a given background noise and participant group. Overall, the more realistic speech task offered a better dynamic range for capturing individual performance and hearing-aid benefit across the range of real-world environments we examined.

VACE-WPE: Virtual Acoustic Channel Expansion Based on Neural Networks for Weighted Prediction Error-Based Speech Dereverberation

Speech dereverberation is an important issue for many real-world speech processing applications. Among the techniques developed, the weighted prediction error (WPE) algorithm has been widely adopted and advanced over the last decade, which blindly cancels out the late reverberation component from the reverberant mixture of microphone signals. In this study, we extend the neural-network-based virtual acoustic channel expansion (VACE) framework for the WPE-based speech dereverberation, a variant of the WPE that we recently proposed to enable the use of dual-channel WPE algorithm in a single-microphone speech dereverberation scenario. Based on the previous study, some ablation studies are conducted regarding the constituents of the VACE-WPE in an offline processing scenario. These studies reveal the characteristics of the system, thereby simplifying the architecture and leading to the introduction of new strategies for training the neural network for the VACE. Experimental results demonstrate that VACE-WPE (our PyTorch implementation and pre-trained models are available from <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/dreadbird06/vace_wpe</uri> ) considerably outperforms its single-channel counterpart in simulated noisy reverberant environments in terms of objective speech quality and is superior to the single-channel WPE as well as several fully neural speech dereverberation methods when employed as the front-end for the far-field automatic speech recognizer.

Towards multilingual end‐to‐end speech recognition for air traffic control

In this work, an end-to-end framework is proposed to achieve multilingual automatic speech recognition (ASR) in air traffic control (ATC) systems. Considering the standard ATC procedure, a recurrent neural network (RNN) based framework is selected to mine the temporal dependencies among speech frames. Facing the distributed feature space caused by the radio transmission, a hybrid feature embedding block is designed to extract high-level representations, in which multiple convolutional neural networks are designed to accommodate different frequency and temporal resolutions. The residual mechanism is performed on the RNN layers to improve the trainability and the convergence. To integrate the multilingual ASR into a single model and relieve the class imbalance, a special vocabulary is designed to unify the pronunciation of the vocabulary in Chinese and English, i.e., pronunciation-oriented vocabulary. The proposed model is optimized by the connectionist temporal classification loss and is validated on a real-world speech corpus (ATCSpeech). A character error rate of 4.4% and 5.9% is achieved for Chinese and English speech, respectively, which outperforms other popular approaches. Most importantly, the proposed approach achieves the multilingual ASR task in an end-to-end manner with considerable high performance.

Self-Supervised Speech Enhancement for Arabic Speech Recognition in Real-World Environments

Mobile speech recognition attracts much attention in the ubiquitous context, however, background noises, speech coding, and transmission errors are prone to corrupt the incoming speech. Therein, building a robust speech recognizer requires the availability of a large number of real-world speech samples. Arabic language, like many other languages, lacks such resources; to overcome this limitation, we propose a speech enhancement step, before the recognition begins. For the speech enhancement purpose, we suggest the use of a deep autoencoder (DAE) algorithm. A two-step procedure is suggested: in the first step, an overcomplete DAE is trained in an unsupervised way, and in the second one, a denoising DAE is trained in a supervised way leveraging the clean speech produced in the previous step. Experimental results performed on a real-life mobile database confirmed the potentials of the proposed approach and show a reduction of the WER (Word Error Rate) of a ubiquitous Arabic speech recognizer. Further experiments show an improvement of the perceptual evaluation of speech quality (PESQ), and the short-time objective intelligibility (STOI) as well.

On correcting the phase bias of GCC in spatially correlated noise fields

Generalized cross correlation (GCC) is arguably the most widely used time difference of arrival (TDOA) estimators. Most studies on GCC assume spatially uncorrelated noises, and focus on the design of weighting functions. However, real world noises tend to be spatially correlated, and such correlation introduces phase bias to the cross phase spectra used in GCC. This paper proposes a simple closed-form maximum likelihood (ML) cross phase spectra estimator for GCC by only assuming the prior knowledge of noise spatial complex coherence function, which can be predicted in theory or measured in advance. TDOA estimation experimental results on real world speech recordings confirm the usefulness of this simple new phase estimator.

Nerdy girls talking gross

This article examines how people view language use in manga (Japanese comic books and graphic novels) through an analysis of posts on a Japanese online bulletin board system. The analysis uncovers three central assumptions regarding texts understood as manga: they lack linguistic sophistication; their linguistic authenticity is problematic; and they negatively impact real-world speech. In the posts, language in manga is often assigned a vaguely negative influence on communication skills, and engagement with manga is foregrounded as a problematic social issue. Beliefs about language in manga parallel commonly held beliefs about language used by otaku ‘nerds/fans’, suggesting that metapragmatic stereotypes have expanded from media users to the media itself. Such criticism often targets women: although otaku are normatively viewed as male, women seen as encroaching masculine forms of engagement with manga may also receive criticism. The article thus contributes to an understanding of how perceptions of language and gender in media are formed.

Audiovisual Enhancement of Speech Perception in Noise by School-Age Children Who Are Hard of Hearing.

The purpose of this study was to examine age- and hearing-related differences in school-age children's benefit from visual speech cues. The study addressed three questions: (1) Do age and hearing loss affect degree of audiovisual (AV) speech enhancement in school-age children? (2) Are there age- and hearing-related differences in the mechanisms underlying AV speech enhancement in school-age children? (3) What cognitive and linguistic variables predict individual differences in AV benefit among school-age children? Forty-eight children between 6 and 13 years of age (19 with mild to severe sensorineural hearing loss; 29 with normal hearing) and 14 adults with normal hearing completed measures of auditory and AV syllable detection and/or sentence recognition in a two-talker masker type and a spectrally matched noise. Children also completed standardized behavioral measures of receptive vocabulary, visuospatial working memory, and executive attention. Mixed linear modeling was used to examine effects of modality, listener group, and masker on sentence recognition accuracy and syllable detection thresholds. Pearson correlations were used to examine the relationship between individual differences in children's AV enhancement (AV-auditory-only) and age, vocabulary, working memory, executive attention, and degree of hearing loss. Significant AV enhancement was observed across all tasks, masker types, and listener groups. AV enhancement of sentence recognition was similar across maskers, but children with normal hearing exhibited less AV enhancement of sentence recognition than adults with normal hearing and children with hearing loss. AV enhancement of syllable detection was greater in the two-talker masker than the noise masker, but did not vary significantly across listener groups. Degree of hearing loss positively correlated with individual differences in AV benefit on the sentence recognition task in noise, but not on the detection task. None of the cognitive and linguistic variables correlated with individual differences in AV enhancement of syllable detection or sentence recognition. Although AV benefit to syllable detection results from the use of visual speech to increase temporal expectancy, AV benefit to sentence recognition requires that an observer extracts phonetic information from the visual speech signal. The findings from this study suggest that all listener groups were equally good at using temporal cues in visual speech to detect auditory speech, but that adults with normal hearing and children with hearing loss were better than children with normal hearing at extracting phonetic information from the visual signal and/or using visual speech information to access phonetic/lexical representations in long-term memory. These results suggest that standard, auditory-only clinical speech recognition measures likely underestimate real-world speech recognition skills of children with mild to severe hearing loss.