Representation Of Speech Research Articles

Deep temporal representation learning for language identification

Language identification (LID) is a key component in downstream tasks. Recently, the self-supervised speech representation learned by Wav2Vec 2.0 (W2V2) has been demonstrated to be very effective for various speech-related tasks. In LID, it is commonly used as a feature extractor for frame-level feature extraction. However, there is currently no effective method for extracting temporal information from frame-level features to enhance the performance of LID systems. To deal with this issue, we propose a LID framework based on deep temporal representation (DTR) learning. First, the W2V2 model is used as a front-end feature extractor. This model can capture contextual representations from continuous raw audio in which temporal dependencies are embedded. Then, a temporal network responsible for learning temporal dependencies is proposed to process the output of W2V2. This temporal network comprises a temporal representation extractor for extracting utterance-level representations and a temporal regularization term to impose constraints on temporal dynamics. Finally, the temporal dependencies are used as utterance-level representations for the subsequent classification. The proposed DTR method is evaluated on the OLR2020 database and compared to other state-of-the-art methods. The results show that the proposed method achieves decent experimental performance on all the three tasks of OLR2020 database.

Multi-scale Information Aggregation for Spoofing Detection

Synthesis artifacts that span scales from small to large are important cues for spoofing detection. However, few spoofing detection models leverage artifacts across different scales together. In this paper, we propose a spoofing detection system built on SincNet and Deep Layer Aggregation (DLA), which leverages speech representations at different levels to distinguish synthetic speech. DLA is totally convolutional with an iterative tree-like structure. The unique topology of DLA makes possible compounding of speech features from convolution layers at different depths, and therefore the local and the global speech representations can be incorporated simultaneously. Moreover, SincNet is employed as the frontend feature extractor to circumvent manual feature extraction and selection. SincNet can learn fine-grained features directly from the input speech waveform, thus making the proposed spoofing detection system end-to-end. The proposed system outperforms the baselines when tested on ASVspoof LA and DF datasets. Notably, our single model surpasses all competing systems in ASVspoof DF competition with an equal error rate (EER) of 13.99%, which demonstrates the importance of multi-scale information aggregation for synthetic speech detection.

Continuous and discrete decoding of overt speech with scalp electroencephalography (EEG).

Neurological disorders affecting speech production adversely impact quality of
life for over 7 million individuals in the US. Traditional speech interfaces like eyetracking
devices and P300 spellers are slow and unnatural for these patients. An
alternative solution, speech Brain-Computer Interfaces (BCIs), directly decodes speech
characteristics, offering a more natural communication mechanism. This research
explores the feasibility of decoding speech features using non-invasive EEG. Nine
neurologically intact participants were equipped with a 63-channel EEG system
with additional sensors to eliminate eye artifacts. Participants read aloud sentences
displayed on a screen selected for phonetic similarity to the English language. Deep
learning models, including Convolutional Neural Networks and Recurrent Neural
Networks with and without attention modules, were optimized with a focus on
minimizing trainable parameters and utilizing small input window sizes for real-time
application. These models were employed for discrete and continuous speech decoding
tasks, achieving statistically significant participant-independent decoding performance
for discrete classes and continuous characteristics of the produced audio signal. A
frequency sub-band analysis highlighted the significance of certain frequency bands
(delta, theta, and gamma) for decoding performance, and a perturbation analysis
was used to identify crucial channels. Assessed channel selection methods did not
significantly improve performance, suggesting a distributed representation of speech
information encoded in the EEG signals. Leave-One-Out training demonstrated
the feasibility of utilizing common speech neural correlates, reducing data collection
requirements from individual participants.

Neuroimaging Findings for the Overnight Consolidation of Learned Non-native Speech Sounds

Abstract Research over the past two decades has documented the importance of sleep to language learning. Sleep has been suggested to play a role in establishing new speech representations as well; however, the neural mechanisms corresponding to sleep-mediated effects on speech perception behavior are unknown. In this study, we trained monolingual English-speaking adults to perceive differences between the Hindi dental vs. retroflex speech contrast in the evening. We examined the blood oxygen level dependent signal using functional magnetic resonance imaging during perceptual tasks on both the trained talker and on an untrained talker shortly after training, and again the next morning. We also employed diffusion tensor imaging to determine if individual differences in white matter structure could predict variability in overnight consolidation. We found greater activity in cortical regions associated with language processing (e.g., left insula) on the second day. Fractional anisotropy values in the anterior thalamic radiation and the uncinate fasciculus was associated with the magnitude of overnight change in perceptual behavior on the generalization (untrained) talker, after controlling for differences in sleep duration and initial learning. Our findings suggest that speech-perceptual information is subject to an overnight transfer of information to the cortex. Moreover, neural structure appears to be linked to individual differences in efficiency of overnight consolidation.

SVQ-MAE: an efficient speech pre-training framework with constrained computational resources

Self-supervised learning for speech pre-training models has achieved remarkable success in acquiring superior speech contextual representations by learning from unlabeled audio, excelling in numerous downstream speech tasks. However, the pre-training of these models necessitates significant computational resources and training duration, presenting a high barrier to entry into the realm of pre-training learning. In our efforts, by amalgamating the resource-efficient benefits of the generative learning model, Masked Auto Encoder, with the efficacy of the vector quantization method in discriminative learning, we introduce a novel pre-training framework: Speech Vector Quantization Masked Auto Encoder (SVQ-MAE). Distinct from the majority of SSL frameworks, which require simultaneous construction of speech contextual representations and mask reconstruction within an encoder-only module, we have exclusively designed a decoupled decoder for pre-training SVQ-MAE. This allows the additional decoupled decoder to undertake the mask reconstruction task solely, reducing the learning complexity of pretext tasks and enhancing the encoder’s efficiency in extracting speech contextual representations. Owing to this innovation, by using only 4 GPUs, SVQ-NAE can achieve high performance comparable to wav2vec 2.0, which requires 64 GPUs for training. In the Speech Processing Universal Performance Benchmark, SVQ-MAE surpasses wav2vec 2.0 in both keyword spotting and emotion recognition tasks. Furthermore, in cross-lingual ASR for Mandarin, upon fine-tuning on AISHELL-1, SVQ-MAE achieves a Character Error Rate of 4.09%, outperforming all supervised ASR models.

Deep CNN for Parkinson’s Disease Classification Using Line Spectral Frequency Images of Sustained Speech Phonation

It is proposed to use Deep Convolution Neural Network (DCNN) which is a good classifier of natural images to learn speech spectrum images of sustained phonation to detect Parkinson’s Disease (PD) as an alternative to the existing feature-based machine learning method. It is shown that the proposed method yields very high accuracy without the need for separate feature computation stage. The speech spectrum representations proposed are Short Time Fourier Transform (STFT) spectrum of size Nx256 and Line Spectral Frequency (LSF) spectrum of size Nx16. LSF reflects the speech production mechanism and it is a novel idea to use LSF spectrum in DCNN to detect PD speech. The spectrum images look like random patterns and the performance is improved when using an additional deeper hidden layer of tampering pattern in the last stage of a fully connected layer. Using a standard PD-sustained phonation dataset the training accuracies achieved are 98.50% and 92.50% for STFT and LSF method, respectively. The validation accuracies achieved are 84.38% for STFT and 100% for LSF. The STFT method results in a sensitivity of 97.05%, a specificity of 88.63%, a precision of 86.84%, an F1-score of 91.66, a false positive rate (FPR) of 11.36%, and a false alarm rate of 12.82%. The LSF method results in a sensitivity 97.05%, a specificity of 95.45%, a precision of 94.28%, an F1-score of 95.65, an FPR of 4.50%, and a false alarm rate of 5.71%. The LSF based method performs better and the performance comparison with the state-of-the-art methods brings out the merits of the LSF spectrum image-based DCNN learning in PD detection using sustained phonation.

Siamese Neural Network for Speech-Based Depression Classification and Severity Assessment

The evaluation of an individual’s mental health and behavioral functioning, known as psychological assessment, is generally conducted by a mental health professional. This process aids in diagnosing mental health conditions, identifying suitable treatment options, and assessing progress during treatment. Currently, national health systems are unable to cope with the constantly growing demand for such services. To address and expedite the diagnosis process, this study suggests an AI-powered tool capable of delivering understandable predictions through the automated processing of the captured speech signals. To this end, we employed a Siamese neural network (SNN) elaborating on standardized speech representations free of domain expert knowledge. Such an SNN-based framework is able to address multiple downstream tasks using the same latent representation. Interestingly, it has been applied both for classifying speech depression as well as assessing its severity. After extensive experiments on a publicly available dataset following a standardized protocol, it is shown to significantly outperform the state of the art with respect to both tasks. Last but not least, the present solution offers interpretable predictions, while being able to meaningfully interact with the medical experts.

Wav2vec2 Without Attention: Do You Need Hopfield Networks for Self-Supervised Learning of Speech Representations?

Wav2vec2 Without Attention: Do You Need Hopfield Networks for Self-Supervised Learning of Speech Representations?

Phoneme Recognition in Korean Singing Voices Using Self-Supervised English Speech Representations

In general, it is difficult to obtain a huge, labeled dataset for deep learning-based phoneme recognition in singing voices. Studying singing voices also offers inherent challenges, compared to speech, because of the distinct variations in pitch, duration, and intensity. This paper proposes a detouring method to overcome this insufficient dataset, and applies it to the recognition of Korean phonemes in singing voices. The method started with pre-training the HuBERT, a self-supervised speech representation model, on a large-scale English corpus. The model was then adapted to the Korean speech domain with a relatively small-scale Korean corpus, in which the Korean phonemes were interpreted as similar English ones. Finally, the speech-adapted model was again trained with a tiny-scale Korean singing voice corpus for speech–singing adaptation. In the final adaptation, melodic supervision was chosen, which utilizes pitch information to improve the performance. For evaluation, the performance on multi-level error rates based on Word Error Rate (WER) was taken. Using the HuBERT-based transfer learning for adaptation improved the phoneme-level error rate of Korean speech by as much as 31.19%. Again, on singing voices by melodic supervision, it improved the rate by 0.55%. The significant improvement in speech recognition underscores the considerable potential of a model equipped with general human voice representations captured from the English corpus that can improve phoneme recognition on less target speech data. Moreover, the musical variation in singing voices is beneficial for phoneme recognition in singing voices. The proposed method could be applied to the phoneme recognition of other languages that have less speech and singing voice corpora.

Reduced neural distinctiveness of speech representations in the middle-aged brain.

Speech perception declines independent of hearing thresholds in middle-age, and the neurobiological reasons are unclear. In line with the age-related neural dedifferentiation hypothesis, we predicted that middle-aged adults show less distinct cortical representations of phonemes and acoustic-phonetic features relative to younger adults. In addition to an extensive audiological, auditory electrophysiological, and speech perceptual test battery, we measured electroencephalographic responses time-locked to phoneme instances (phoneme-related potential; PRP) in naturalistic, continuous speech and trained neural network classifiers to predict phonemes from these responses. Consistent with age-related neural dedifferentiation, phoneme predictions were less accurate, more uncertain, and involved a broader network for middle-aged adults compared with younger adults. Representational similarity analysis revealed that the featural relationship between phonemes was less robust in middle-age. Electrophysiological and behavioral measures revealed signatures of cochlear neural degeneration (CND) and speech perceptual deficits in middle-aged adults relative to younger adults. Consistent with prior work in animal models, signatures of CND were associated with greater cortical dedifferentiation, explaining nearly a third of the variance in PRP prediction accuracy together with measures of acoustic neural processing. Notably, even after controlling for CND signatures and acoustic processing abilities, age-group differences in PRP prediction accuracy remained. Overall, our results reveal "fuzzier" phonemic representations, suggesting that age-related cortical neural dedifferentiation can occur even in middle-age and may underlie speech perceptual challenges, despite a normal audiogram.

Analyzing Wav2Vec 1.0 Embeddings for Cross-Database Parkinson's Disease Detection and Speech Features Extraction.

Advancements in deep learning speech representations have facilitated the effective use of extensive unlabeled speech datasets for Parkinson's disease (PD) modeling with minimal annotated data. This study employs the non-fine-tuned wav2vec 1.0 architecture to develop machine learning models for PD speech diagnosis tasks, such as cross-database classification and regression to predict demographic and articulation characteristics. The primary aim is to analyze overlapping components within the embeddings on both classification and regression tasks, investigating whether latent speech representations in PD are shared across models, particularly for related tasks. Firstly, evaluation using three multi-language PD datasets showed that wav2vec accurately detected PD based on speech, outperforming feature extraction using mel-frequency cepstral coefficients in the proposed cross-database classification scenarios. In cross-database scenarios using Italian and English-read texts, wav2vec demonstrated performance comparable to intra-dataset evaluations. We also compared our cross-database findings against those of other related studies. Secondly, wav2vec proved effective in regression, modeling various quantitative speech characteristics related to articulation and aging. Ultimately, subsequent analysis of important features examined the presence of significant overlaps between classification and regression models. The feature importance experiments discovered shared features across trained models, with increased sharing for related tasks, further suggesting that wav2vec contributes to improved generalizability. The study proposes wav2vec embeddings as a next promising step toward a speech-based universal model to assist in the evaluation of PD.

Efficiency-oriented approaches for self-supervised speech representation learning

Efficiency-oriented approaches for self-supervised speech representation learning

Auditory cortex encodes lipreading information through spatially distributed activity

Watching a speaker's face improves speech perception accuracy. This benefit is enabled, in part, by implicit lipreading abilities present in the general population. While it is established that lipreading can alter the perception of a heard word, it is unknown how these visual signals are represented in the auditory system or how they interact with auditory speech representations. One influential, but untested, hypothesis is that visual speech modulates the population-coded representations of phonetic and phonemic features in the auditory system. This model is largely supported by data showing that silent lipreading evokes activity in the auditory cortex, but these activations could alternatively reflect general effects of arousal or attention or the encoding of non-linguistic features such as visual timing information. This gap limits our understanding of how vision supports speech perception. To test the hypothesis that the auditory system encodes visual speech information, we acquired functional magnetic resonance imaging (fMRI) data from healthy adults and intracranial recordings from electrodes implanted in patients with epilepsy during auditory and visual speech perception tasks. Across both datasets, linear classifiers successfully decoded the identity of silently lipread words using the spatial pattern of auditory cortex responses. Examining the time course of classification using intracranial recordings, lipread words were classified at earlier time points relative to heard words, suggesting a predictive mechanism for facilitating speech. These results support a model in which the auditory system combines the joint neural distributions evoked by heard and lipread words to generate a more precise estimate of what was said.

Classifying coherent versus nonsense speech perception from EEG using linguistic speech features

When a person listens to natural speech, the relation between features of the speech signal and the corresponding evoked electroencephalogram (EEG) is indicative of neural processing of the speech signal. Using linguistic representations of speech, we investigate the differences in neural processing between speech in a native and foreign language that is not understood. We conducted experiments using three stimuli: a comprehensible language, an incomprehensible language, and randomly shuffled words from a comprehensible language, while recording the EEG signal of native Dutch-speaking participants. We modeled the neural tracking of linguistic features of the speech signals using a deep-learning model in a match-mismatch task that relates EEG signals to speech, while accounting for lexical segmentation features reflecting acoustic processing. The deep learning model effectively classifies coherent versus nonsense languages. We also observed significant differences in tracking patterns between comprehensible and incomprehensible speech stimuli within the same language. It demonstrates the potential of deep learning frameworks in measuring speech understanding objectively.

Speech self-supervised representations benchmarking: A case for larger probing heads

Self-supervised learning (SSL) leverages large datasets of unlabeled speech to reach impressive performance with reduced amounts of annotated data. The high number of proposed approaches fostered the emergence of comprehensive benchmarks that evaluate their performance on a set of downstream tasks exploring various aspects of the speech signal. However, while the number of considered tasks has been growing, most proposals rely upon a single downstream architecture that maps the frozen SSL representations to the task labels. This study examines how benchmarking results are affected by changes in the probing head architecture. Interestingly, we found that altering the downstream architecture structure leads to significant fluctuations in the performance ranking of the evaluated models. Against common practices in speech SSL benchmarking, we evaluate larger-capacity probing heads, showing their impact on performance, inference costs, generalization, and multi-level feature exploitation.

Perceptual anchoring: Children with dyslexia benefit less than controls from contextual repetitions in speech processing

ObjectivesIndividuals with dyslexia perceive and utilize statistical features in the auditory input deficiently. The present study investigates whether affected children also benefit less from repeating context tones as perceptual anchors for subsequent speech processing. MethodsIn an event-related potential study, eleven-year-old children with dyslexia (n = 21) and without dyslexia (n = 20) heard syllable pairs, with the first syllable either receiving a constant pitch (anchor) or variable pitch (no-anchor), while second syllables were identical across conditions. ResultsChildren with and without dyslexia showed smaller auditory P2 responses to constant-pitch versus variable-pitch first syllables, while only control children additionally showed smaller N1 and faster P1 responses. This suggests less automatic processing of anchor repetitions in dyslexia. For the second syllables, both groups showed faster P2 responses following anchor than no-anchor first syllables, but only controls additionally showed smaller P2 responses. ConclusionsChildren with and without dyslexia show differences in anchor effects. While both groups seem to allocate less attention to speech stimuli after contextual repetitions, children with dyslexia display less facilitation in speech processing from acoustic anchors. SignificanceAltered anchoring in the linguistic domain may contribute to the difficulties of individuals with dyslexia in establishing long-term representations of speech.

Functional mapping of movement and speech using task-based electrophysiological changes in stereoelectroencephalography.

Stereoelectroencephalography (SEEG) has become the predominant method for intracranial seizure localization. When imaging, semiology, and scalp EEG findings are not in full agreement or definitively localizing, implanted SEEG recordings are used to test candidate seizure onset zones (SOZs). Discovered SOZs may then be targeted for resection, laser ablation, or neurostimulation. If an SOZ is eloquent, resection and ablation are both contraindicated, so identifying functional representation is crucial for therapeutic decision-making. The authors present a novel functional brain mapping technique that utilizes task-based electrophysiological changes in SEEG during behavioral tasks and test this in pediatric and adult patients. SEEG was recorded in 20 patients with epilepsy who ranged in age from 6 to 39 years (12 female, 18 of 20 patients < 21 years of age) and underwent implanted monitoring to identify seizure onset. Each performed 1) visually cued simple repetitive movements of the hand, foot, or tongue while electromyography was recorded; and 2) simple picture-naming or verb-generation speech tasks while audio was recorded. Broadband changes in the power spectrum of the SEEG recording were compared between behavior and rest. Electrophysiological functional mapping of movement and/or speech areas was completed in all 20 patients. Eloquent representation was identified in both cortex and white matter and generally corresponded to classically described functional anatomical organization as well as other clinical mapping results. Robust maps of brain activity were identified in healthy brain, regions of developmental or acquired structural abnormality, and SOZs. Task-based electrophysiological mapping using broadband changes in the SEEG signal reliably identifies movement and speech representation in pediatric and adult epilepsy patients.

Audio-guided self-supervised learning for disentangled visual speech representations

Audio-guided self-supervised learning for disentangled visual speech representations

Race, legislative speech, and symbolic representation in Congress

AbstractWe know little about the extent to which racial minorities are symbolically represented by members of Congress. This stands in contrast to a wealth of research analyzing the extent to which minorities are substantively and descriptively represented. This article provides the most comprehensive analysis of symbolic representation to date. Using data on legislators’ speech from 105,875 newsletters and 620,838 floor speeches, I find that White legislators of both parties are more likely to symbolically represent Blacks, Hispanics, and Asians if those groups are more populous in their constituency. However, these effects only hold cross‐sectionally; using a difference‐in‐differences setup from redistricting shocks, I find that there is little within‐legislator variation in speech patterns as their constituencies change. Lastly, I show that, unlike on the symbolic dimension, legislators’ substantive representation is not influenced by group size. I conclude that White legislators are symbolically responsive to their constituents’ identities in their speech patterns.

TadaStride: Using time adaptive strides in audio data for effective downsampling

In this paper, we introduce a new downsampling method for audio data called TadaStride, which can adaptively adjust the downsampling ratios across an audio data instance. Unlike previous methods using a fixed downsampling ratio, TadaStride can preserve more information from task-relevant parts of a data instance by using smaller strides for those parts and larger strides for less relevant parts. Additionally, we also introduce TadaStride-F, which is developed as a more efficient version of TadaStride while maintaining minimal performance loss. In experiments, we evaluate our TadaStride, primarily focusing on a range of audio processing tasks. Firstly, in audio classification experiments, TadaStride and TadaStride-F outperform other widely used standard downsampling methods, even with comparable memory and time usage. Furthermore, through various analyses, we provide an understanding of how TadaStride learns effective adaptive strides and how it leads to improved performance. In addition, through additional experiments on automatic speech recognition and discrete speech representation learning, we demonstrate that TadaStride and TadaStride-F consistently outperform other downsampling methods and examine how the adaptive strides are learned in these tasks.