Speech Segments Research Articles

Conversational Short-Phrase Speaker Diarization via Self-Adjusting Speech Segmentation and Embedding Extraction

Conversational Short-Phrase Speaker Diarization via Self-Adjusting Speech Segmentation and Embedding Extraction

Classification of dementia from spoken speech using feature selection and the bag of acoustic words model

<p>Memory disorders and dementia are a central factor in the decline of functioning and daily activities in older individuals. The workload related to standardized speech tests in clinical settings has led to a growing emphasis on developing automatic machine learning techniques for analyzing naturally spoken speech. This study presented a bag of acoustic words approach for distinguishing dementia patients from control individuals based on audio speech recordings. In this approach, each individual's speech was segmented into voiced periods, and these segments were characterized by acoustic features using the open-source openSMILE library. Word histogram representations were formed from the characterized speech segments of each speaker, which were used for classifying subjects. The formation of word histograms involved a clustering phase where feature vectors were quantized. It is well-known that partitional clustering involves instability in clustering results due to the selection of starting points, which can cause variability in classification outcomes. This study aimed to address instability by utilizing robust K-spatial-medians clustering, efficient K-means$ ++ $ clustering initialization, and selecting the smallest clustering error from repeated clusterings. Additionally, the study employed feature selection based on the Wilcoxon signed-rank test to achieve computational efficiency in the methods. The results showed that it is possible to achieve a consistent 75% classification accuracy using only twenty-five features, both with the external ADReSS 2020 test data and through leave-one-subject-out cross-validation of the entire dataset. The results rank at the top compared to international research, where the same dataset and only acoustic features have been used to diagnose patients.</p>

Contributions of listening effort and intelligibility to cortical tracking of speech in adverse listening conditions

Cortical tracking of speech is vital for speech segmentation and is linked to speech intelligibility. However, there is no clear consensus as to whether reduced intelligibility leads to a decrease or an increase in cortical speech tracking, warranting further investigation of the factors influencing this relationship. One such factor is listening effort, defined as the cognitive resources necessary for speech comprehension, and reported to have a strong negative correlation with speech intelligibility. Yet, no studies have examined the relationship between speech intelligibility, listening effort, and cortical tracking of speech. The aim of the present study was thus to examine these factors in quiet and distinct adverse listening conditions. Forty-nine normal hearing adults listened to sentences produced casually, presented in quiet and two adverse listening conditions: cafeteria noise and reverberant speech. Electrophysiological responses were registered with electroencephalogram, and listening effort was estimated subjectively using self-reported scores and objectively using pupillometry. Results indicated varying impacts of adverse conditions on intelligibility, listening effort, and cortical tracking of speech, depending on the preservation of the speech temporal envelope. The more distorted envelope in the reverberant condition led to higher listening effort, as reflected in higher subjective scores, increased pupil diameter, and stronger cortical tracking of speech in the delta band. These findings suggest that using measures of listening effort in addition to those of intelligibility is useful for interpreting cortical tracking of speech results. Moreover, reading and phonological skills of participants were positively correlated with listening effort in the cafeteria condition, suggesting a special role of expert language skills in processing speech in this noisy condition. Implications for future research and theories linking atypical cortical tracking of speech and reading disorders are further discussed.

Speech and beatboxing cooperate and compromise in beatrhyming

Beatrhyming is a type of multi-vocalism in which an individual beatboxes and speaks (i.e., sings or raps) at the same time by interweaving beatboxing sounds and speech sounds within words and phrases. The measurements in this case study of a beatrhyming performance focus on one-to-one sound “replacements” in which a beatboxing sound is integrated into a word by taking the place of a speech sound. The analysis unfolds in two parts: first, a count of how many times beatboxing sounds used in place of speech sounds matched the intended speech sounds for vocal tract constrictor and constriction degree; and second, an assessment of whether the beatboxing sound patterns in beatrhyming (beatboxing with simultaneous lyrics) have the same degree of rhythmic structure as the beatboxing sound patterns in beatboxing (without lyrics). Despite having disparate aims, the separate speech and beatboxing systems work together to create a well-organized combined behavior. Speech tasks (i.e., communicating the linguistic message of the lyrics) are achieved in beatrhyming by replacing some speech sounds with beatboxing sounds that match the speech segment in vocal tract constrictor and in manner/constriction degree. Beatboxing tasks (i.e., establishing a musical rhythm) are achieved through the inviolable use of Outward K Snares {K} on the backbeat. Achieving both of these aims in the same performance requires flexibility and compromise between the speech and beatboxing systems. In addition to providing the first scientific description and analysis of beatrhyming, this article shows how beatrhyming offers new insight for phonological theories built to describe spoken language.

Improving Text-Independent Forced Alignment to Support Speech-Language Pathologists with Phonetic Transcription.

Problem: Phonetic transcription is crucial in diagnosing speech sound disorders (SSDs) but is susceptible to transcriber experience and perceptual bias. Current forced alignment (FA) tools, which annotate audio files to determine spoken content and its placement, often require manual transcription, limiting their effectiveness. Method: We introduce a novel, text-independent forced alignment model that autonomously recognises individual phonemes and their boundaries, addressing these limitations. Our approach leverages an advanced, pre-trained wav2vec 2.0 model to segment speech into tokens and recognise them automatically. To accurately identify phoneme boundaries, we utilise an unsupervised segmentation tool, UnsupSeg. Labelling of segments employs nearest-neighbour classification with wav2vec 2.0 labels, before connectionist temporal classification (CTC) collapse, determining class labels based on maximum overlap. Additional post-processing, including overfitting cleaning and voice activity detection, is implemented to enhance segmentation. Results: We benchmarked our model against existing methods using the TIMIT dataset for normal speakers and, for the first time, evaluated its performance on the TORGO dataset containing SSD speakers. Our model demonstrated competitive performance, achieving a harmonic mean score of 76.88% on TIMIT and 70.31% on TORGO. Implications: This research presents a significant advancement in the assessment and diagnosis of SSDs, offering a more objective and less biased approach than traditional methods. Our model's effectiveness, particularly with SSD speakers, opens new avenues for research and clinical application in speech pathology.

Using Spontaneous Speech Recognition as a Biomarker to Distinguish Dementia Patients

AbstractBackgroundDigital biomarkers such as individual speech recordings, are a cost‐effective tool for assessing cognitive status using linguistic and acoustic content. The purpose of this study is to investigate whether Natural Language Processing (NLP) methods and audio feature extraction can distinguish spontaneous speech outputs in dementia patients from a healthy population.Method240 audio samples and transcripts of clinician‐participant dialogues involving the Cookie‐Theft visual description were acquired from the DementiaBank Pitt corpus (Dementia group (DG) n = 169; healthy controls (CG) n = 71). Lexical variety, syntactic complexity, salience of information, disfluency patterns, and sentiment analysis were evaluated. Sound synthesis and an acoustic R analysis package were used to filter and examine acoustic properties of recorded audio such as silence and speech segmentation, pitch tracking, and frequency modulation of vibrato or jitter. DG and CG were compared using non‐parametric Kruskal‐Wallis tests, Dunn’s multiple comparison, and Mann‐Whitney U‐test in the statistical analysis. In comparison to CG, we hypothesized that DG would differ in levels of lexical diversity, grammatical complexity, and overall salience.ResultOverall, text analysis showed no variations in disfluencies between groups, but there were significant differences in lexical diversity, syntactic complexity and reference rate to reality (p<0.0001). Additionally, there were no appreciable differences in the frequency of salient information between the two groups, however the frequency of non‐salient information and disfluencies was considerably higher in the DG (p<0.05). Analysis of acoustic features revealed that DG had longer average speech duration and speech segment durations compared to CG (p<0.001).. On the other hand, the rate of change of acoustic characteristics including pitch, mean entropy, and harmonics‐to‐noise ratios did not significantly differ between DG and CG.ConclusionIn conclusion, this paradigm reveals that, in comparison to healthy controls, dementia patients exhibited increased lexical diversity yet lower linguistic complexity, and a shift towards characterizing non‐salient objects in given stimuli, as well as longer segments of speech. Further examination of the rhythmic and acoustic characteristics of speech may provide a comprehensive framework towards defining speech as a potential digital biomarker for dementia risk, and provide applications to regions experiencing increasing socioeconomic disparities.

Plug-and-Play Microphones for Recording Speech and Voice with Smart Devices

Introduction: Smart devices are widely available and capable of quickly recording and uploading speech segments for health-related analysis. The switch from laboratory recordings with professional-grade microphone setups to remote, smart device-based recordings offers immense potential for the scalability of voice assessment. Yet, a growing body of literature points to a wide heterogeneity among acoustic metrics for their robustness to variation in recording devices. The addition of consumer-grade plug-and-play microphones has been proposed as a possible solution. The aim of our study was to assess if the addition of consumer-grade plug-and-play microphones increases the acoustic measurement agreement between ultra-portable devices and a reference microphone. Methods: Speech was simultaneously recorded by a reference high-quality microphone commonly used in research and by two configurations with plug-and-play microphones. Twelve speech-acoustic features were calculated using recordings from each microphone to determine the agreement intervals in measurements between microphones. Agreement intervals were then compared to expected deviations in speech in various neurological conditions. Each microphone’s response to speech and to silence was characterized through acoustic analysis to explore possible reasons for differences in acoustic measurements between microphones. The statistical differentiation of two groups, neurotypical and people with multiple sclerosis, using metrics from each tested microphone was compared to that of the reference microphone. Results: The two consumer-grade plug-and-play microphones favored high frequencies (mean center of gravity difference ≥ +175.3 Hz) and recorded more noise (mean difference in signal to noise ≤ −4.2 dB) when compared to the reference microphone. Between consumer-grade microphones, differences in relative noise were closely related to distance between the microphone and the speaker’s mouth. Agreement intervals between the reference and consumer-grade microphones remained under disease-expected deviations only for fundamental frequency (f0, agreement interval ≤0.06 Hz), f0 instability (f0 CoV, agreement interval ≤0.05%), and tracking of second formant movement (agreement interval ≤1.4 Hz/ms). Agreement between microphones was poor for other metrics, particularly for fine timing metrics (mean pause length and pause length variability for various tasks). The statistical difference between the two groups of speakers was smaller with the plug-and-play than with the reference microphone. Conclusion: Measurement of f0 and F2 slope was robust to variation in recording equipment, while other acoustic metrics were not. Thus, the tested plug-and-play microphones should not be used interchangeably with professional-grade microphones for speech analysis. Plug-and-play microphones may assist in equipment standardization within speech studies, including remote or self-recording, possibly with small loss in accuracy and statistical power as observed in the current study.

A Novel Approach for Speech Recognition of Malayalam Using Enhanced MFCC Features

The study Investigates into the realm of speech recognition, particularly focusing on the evaluation of Hidden Markov Models (HMM), leveraging the MalayalamVoice dataset. The research scrutinizes the performance of HMM concerning word error rate (WER) and accuracy across varied word lengths, revealing a consistent trend of increased WER and decreased accuracy with longer utterances. This underscores the challenges inherent in accurately transcribing extended speech segments, accentuating the necessity for algorithmic enhancements. Moreover, analyses across diverse datasets and noisy environments underscore the criticality of comprehending dataset characteristics for refining recognition algorithms. Additionally, comparisons of different feature extraction methods elucidate the efficacy of Enhanced MFCC, particularly for shorter word lengths. However, as the word length extends, the distinctions between extraction methods diminish, highlighting the multifaceted nature of speech recognition. Overall, this study underscores the intricacies involved in speech recognition and the imperative of algorithmic refinements for augmenting accuracy, especially in practical scenarios.

Diffusion tensor imaging-based multi-fiber tracking reconstructions can regionally differentiate phonetic versus prosodic subtypes of progressive apraxia of speech

Two subtypes of progressive apraxia of speech (PAOS) have been recognized: phonetic PAOS (PAOS_ph) where speech output is dominated by distorted sound substitutions and prosodic PAOS (PAOS_pr) which is dominated by segmented speech. We investigate whether these PAOS subtypes have different white matter microstructural abnormalities measured by diffusion tensor tractography. Thirty-three patients with PAOS (21 PAOS_ph and 12 PAOS_pr) and 19 healthy controls were recruited by the Neurodegenerative Research Group (NRG) and underwent diffusion MRI. Using a whole-brain tractography approach, fractional anisotropy (FA) and mean diffusivity (MD) were extracted for cortico-cortical, cortico-subcortical, cortical-projection, and cerebello-cortical white matter tracts. A hierarchical linear model was applied to assess tract-level FA and MD across groups. Both PAOS_ph and PAOS_pr showed degeneration of cortico-cortical, cortico-subcortical, cortical-projection, and cerebello-cortical white matter tracts compared to controls. However, degeneration of the body of corpus callosum, superior thalamic radiation, and superior cerebellar peduncle was greater in PAOS_pr compared to PAOS_ph, and degeneration of the inferior segment of the superior longitudinal fasciculus (SLF) was greater in PAOS_ph compared to PAOS_pr. Worse parkinsonism correlated with greater degeneration of cortico-cortical and cortico-subcortical tracts in PAOS_ph. Apraxia of speech articulatory error score correlated with degeneration of the superior cerebellar peduncle tracts in PAOS_pr. Phonetic and prosodic PAOS involve the compromise of a similar network of tracts, although there are connectivity differences between types. Whereas clinical parameters are the current gold standard to distinguish PAOS subtypes, our results allege the use of DTI-based tractography as a supplementary method to investigate such variants.

Insights into L2 connected speech segmentation: A gating experiment with listeners of different English proficiency levels

The relative contribution of bottom-up (i.e. acoustic-phonetic) and top-down (i.e. contextual) cues for successful L2 online segmentation is still a matter of debate. This study used the gating paradigm to investigate the segmentation processes of adult L2 English listeners with different proficiency levels, by looking at the type of cues they exploit and how they revise their hypotheses as connected speech is progressively revealed. Twenty-one French and Tunisian undergraduates were selected from a larger pool (n=226) and identified as skilled (n=11) and unskilled (n=10) listeners based on their scores on standardized English listening and vocabulary tests. Descriptive statistics, analysis of variance and qualitative analysis were performed on the obtained data. Overall, this study provides supporting L2 evidence for the hierarchical nature of the multiple speech segmentation cues (Mattys et al., 2005). The results indicated an early effect of context on segmentation that was independent of L2 proficiency when the context is constraining. In non-constraining contexts, successful segmentation is delayed for both groups with L2 unskilled listeners needing far more bottom-up information to process input and revise their segmentation hypotheses. We conclude that, in online L2 speech segmentation, what distinguishes proficient from non-proficient listeners is their efficient processing of bottom-up cues. Pedagogical implications are provided hoping to help L2 English teachers (and materials developers) focus on bottom-up training to improve their learners’ real-time comprehension competence.

RECOGNITION OF ORAL SPEECH ACCORDING TO MEG DATA BY COVARIANCE FILTERS

Speech recognition based on EEG and MEG data is the first step in the development of BCI and AI systems for their further use in inner speech decoding. Great advances in this direction have been made using ECoG and stereo-EEG. At the same time, there are few works on this topic on the analysis of data obtained by non-invasive methods of recording brain activity. Our approach is based on the evaluation of connections in the space of sensors with the identification of a pattern of MEG connectivity specific for a given segment of speech. We tested our method on 7 subjects. In all cases, our processing pipeline was quite reliable and worked either without recognition errors or with a small number of errors. After “training”, the algorithm is able to recognise a fragment of oral speech with a single presentation. For recognition, we used segments of the MEG recording 50–1200 ms from the beginning of the sound of the word. For high-quality recognition, a segment of at least 600 ms was required. Intervals longer than 1200 ms worsened the recognition quality. Bandpass filtering of the MEG showed that the quality of recognition is equally effective in the entire frequency range. Some decrease in the level of recognition is observed only in the range of 9–14 Hz.

Sentence recognition with modulation-filtered speech segments for younger and older adults: Effects of hearing impairment and cognition.

This study investigated word recognition for sentences temporally filtered within and across acoustic-phonetic segments providing primarily vocalic or consonantal cues. Amplitude modulation was filtered at syllabic (0-8 Hz) or slow phonemic (8-16 Hz) rates. Sentence-level modulation properties were also varied by amplifying or attenuating segments. Participants were older adults with normal or impaired hearing. Older adult speech recognition was compared to groups of younger normal-hearing adults who heard speech unmodified or spectrally shaped with and without threshold matching noise that matched audibility to hearing-impaired thresholds. Participants also completed cognitive and speech recognition measures. Overall, results confirm the primary contribution of syllabic speech modulations to recognition and demonstrate the importance of these modulations across vowel and consonant segments. Group differences demonstrated a hearing loss-related impairment in processing modulation-filtered speech, particularly at 8-16 Hz. This impairment could not be fully explained by age or poorer audibility. Principal components analysis identified a single factor score that summarized speech recognition across modulation-filtered conditions; analysis of individual differences explained 81% of the variance in this summary factor among the older adults with hearing loss. These results suggest that a combination of cognitive abilities and speech glimpsing abilities contribute to speech recognition in this group.

Assessment of Tracks of Resonance Frequencies of the Vocal Tract

A new method for estimating formant frequency tracks of the vocal tract for arbitrary speech segments is proposed. The method uses the ratio of two Fourier transforms of a speech signal with special exponential-type windows depending on some parameter. This ratio is used for specific points in time and is considered as a function of frequency and parameter. By analyzing, for several parameter values, the distribution of minimum points (in terms of frequency) for the phase of this ratio and/or a similar distribution of extreme points for its amplitude, it is possible to estimate formant frequencies from the peaks of these distributions. A mathematical study is presented that substantiates this approach. A series of numerical experiments were carried out on the processing of synthetic and real speech signals, which confirmed the performance capabilities of the proposed formant evaluation method. In particular, in experiments with synthesized vowels, it was found that the error in estimating their resonance frequencies is small and stable with respect to additive noise up to a signal-to-noise ratio of 5 dB. For real speech, the method makes it possible to calculate the formant frequency tracks for both sounds with vocal excitation and for voiceless fricatives, aspirated plosives, and whispered speech.

SUPRASEGMENTAL PHONOLOGICAL UNITS IN LINGUODIDACTIC TRAINING OF FUTURE UKRAINIAN LANGUAGE AND LITERATURE TEACHERS

This publication examines linguistic and linguodidactic sources on the study and mastery of suprasegmental phonetic units, including stress, intonation, pauses, and prosody. The research is conducted using methods of analysis, synthesis, and comparison. The acoustic, perceptual, and functional characteristics of these suprasegmental features are explored. It is revealed that in the field of linguodidactics, the information regarding suprasegmental phonetic elements is often simplified, with limited consideration given to the theoretical and methodological foundations of prosody and intonation. Furthermore, the connection between these suprasegmental features and the accentual subsystem of phonetics is not consistently traced.The content of the case study titled “Suprasegmental Phonetic Units in the Methodological System of Teaching Phonetics to Future Ukrainian Literature Teachers” has been developed and made open for discussion, critical analysis, and enhancement within the interactive technology of the “Linguodidactic Studio”. The content of the case study aims at facilitating a comprehensive understanding of the linguistic material concerning the suprasegmental level of language, which represents both prosody and intonation and is viewed differently by researchers. The linguistic units at this level include accentemes, prosodemes, and intonemes. It has been observed that in spoken language, accentemes correlate with stress, while prosodemes and intonemes correlate with stress, tempo, timbre, rhythm, tone, melody, loudness, and pauses. Considering this, the case study includes research on the interconnectedness of the accentual system of the Ukrainian language with intonation and prosody, as well as the influence of stress on the functioning of a combination of prosodic and intonational components that overlay segmental phonetic units such as sounds, syllables, consonant clusters, rhythmic structures, phrases, and sentences. Through an analysis of the locational sphere of intonation (phrase segments) and prosody (all speech segments, from syllables to text), their relationship has been clarified. It is argued that suprasegmental units in linguodidactics can be studied within the structure of the accentological sub-level.
 Keywords: Suprasegmental Units of Phonetics; Stress; Prosody; Intonation; Accentological Sublevel; Teacher-Wordist; Linguistic Case; “Linguodidactic Studies”.

DYNAMIC PECULIAITIES OF MARITIME COMMANDS IN THE ENGLISH SHIPPING DISCOURSE AND FEATURE FILM DISCOURSE

The article is dedicated to the study of the dynamic features of the prosodic organization of English maritime commands used in two types of discourse: authentic shipping discourse and feature film discourse. The research material included audio and video recordings of 120 maritime commands with a total duration of 72 minutes. Audio and video recordings of 60 implementations of English maritime commands were used to investigate commands in the authentic shipping discourse, while video and soundtracks from 10 English-language films containing 60 commands were used to investigate maritime commands spoken by the characters in English-language feature films. The work uses two types of analysis, namely: perceptual analysis of speech segments and instrumental analysis using the PRAAT 6.1.05 software package. The results of a comparative analysis of the prosodic features of authentic maritime commands with maritime commands extracted from British and American films indicate that they differ significantly in a number of dynamic parameters, including different loudness and intensity range; localization of intensity maxima within the syntagm; the coincidence of the maxima of the fundamental frequency and the intensity within the syntagm. A common feature of maritime commands in the two studied types of discourse is the most frequent use of high and volume moderate during their utterance. The study proves that maritime commands in English-language films are characterized by a wider intensity range compared to authentic maritime commands; the maximum of intensity in the syntagm is most frequently localized on the nuclear syllable, and the percentage of the localization on the scale is the lowest; the highest coincidence of the maximum of the fundamental frequency and the intensity also falls on the nuclear syllable of the syntagm.

An Automatic Speaker Clustering Pipeline for the Air Traffic Communication Domain

In air traffic management (ATM), voice communications are critical for ensuring the safe and efficient operation of aircraft. The pertinent voice communications—air traffic controller (ATCo) and pilot—are usually transmitted in a single channel, which poses a challenge when developing automatic systems for air traffic management. Speaker clustering is one of the challenges when applying speech processing algorithms to identify and group the same speaker among different speakers. We propose a pipeline that deploys (i) speech activity detection (SAD) to identify speech segments, (ii) an automatic speech recognition system to generate the text for audio segments, (iii) text-based speaker role classification to detect the role of the speaker—ATCo or pilot in our case—and (iv) unsupervised speaker clustering to create a cluster of each individual pilot speaker from the obtained speech utterances. The speech segments obtained by SAD are input into an automatic speech recognition (ASR) engine to generate the automatic English transcripts. The speaker role classification system takes the transcript as input and uses it to determine whether the speech was from the ATCo or the pilot. As the main goal of this project is to group the speakers in pilot communication, only pilot data acquired from the classification system is employed. We present a method for separating the speech parts of pilots into different clusters based on the speaker’s voice using agglomerative hierarchical clustering (AHC). The performance of the speaker role classification and speaker clustering is evaluated on two publicly available datasets: the ATCO2 corpus and the Linguistic Data Consortium Air Traffic Control Corpus (LDC-ATCC). Since the pilots’ real identities are unknown, the ground truth is generated based on logical hypotheses regarding the creation of each dataset, timing information, and the information extracted from associated callsigns. In the case of speaker clustering, the proposed algorithm achieves an accuracy of 70% on the LDC-ATCC dataset and 50% on the more noisy ATCO2 dataset.

Enhancing Parkinson's disease severity assessment through voice-based wavelet scattering, optimized model selection, and weighted majority voting

Parkinson's disease (PD) is a neurodegenerative disorder characterized by motor and non-motor symptoms that significantly impact an individual's quality of life. Voice changes have shown promise as early indicators of PD, making voice analysis a valuable tool for early detection and intervention. This study aims to assess and detect the severity of PD through voice analysis using the mobile device voice recordings dataset. The dataset consisted of recordings from PD patients at different stages of the disease and healthy control subjects. A novel approach was employed, incorporating a voice activity detection algorithm for speech segmentation and the wavelet scattering transform for feature extraction. A Bayesian optimization technique is used to fine-tune the hyperparameters of seven commonly used classifiers and optimize the performance of machine learning classifiers for PD severity detection. AdaBoost and K-nearest neighbor consistently demonstrated superior performance across various evaluation metrics among the classifiers. Furthermore, a weighted majority voting (WMV) technique is implemented, leveraging the predictions of multiple models to achieve a near-perfect accuracy of 98.62%, improving classification accuracy. The results highlight the promising potential of voice analysis in PD diagnosis and monitoring. Integrating advanced signal processing techniques and machine learning models provides reliable and accessible tools for PD assessment, facilitating early intervention and improving patient outcomes. This study contributes to the field by demonstrating the effectiveness of the proposed methodology and the significant role of WMV in enhancing classification accuracy for PD severity detection.

Decoding speech perception from non-invasive brain recordings

Decoding speech from brain activity is a long-awaited goal in both healthcare and neuroscience. Invasive devices have recently led to major milestones in this regard: deep-learning algorithms trained on intracranial recordings can now start to decode elementary linguistic features such as letters, words and audio-spectrograms. However, extending this approach to natural speech and non-invasive brain recordings remains a major challenge. Here we introduce a model trained with contrastive learning to decode self-supervised representations of perceived speech from the non-invasive recordings of a large cohort of healthy individuals. To evaluate this approach, we curate and integrate four public datasets, encompassing 175 volunteers recorded with magneto-encephalography or electro-encephalography while they listened to short stories and isolated sentences. The results show that our model can identify, from 3 seconds of magneto-encephalography signals, the corresponding speech segment with up to 41% accuracy out of more than 1,000 distinct possibilities on average across participants, and with up to 80% in the best participants—a performance that allows the decoding of words and phrases absent from the training set. The comparison of our model with a variety of baselines highlights the importance of a contrastive objective, pretrained representations of speech and a common convolutional architecture simultaneously trained across multiple participants. Finally, the analysis of the decoder’s predictions suggests that they primarily depend on lexical and contextual semantic representations. Overall, this effective decoding of perceived speech from non-invasive recordings delineates a promising path to decode language from brain activity, without putting patients at risk of brain surgery.

PulmoListener

Prior work has shown the utility of acoustic analysis in controlled settings for assessing chronic obstructive pulmonary disease (COPD) --- one of the most common respiratory diseases that impacts millions of people worldwide. However, such assessments require active user input and may not represent the true characteristics of a patient's voice. We propose PulmoListener, an end-to-end speech processing pipeline that identifies segments of the patient's speech from smartwatch audio collected during daily living and analyzes them to classify COPD symptom severity. To evaluate our approach, we conducted a study with 8 COPD patients over 164 ± 92 days on average. We found that PulmoListener achieved an average sensitivity of 0.79 ± 0.03 and a specificity of 0.83 ± 0.05 per patient when classifying their symptom severity on the same day. PulmoListener can also predict the severity level up to 4 days in advance with an average sensitivity of 0.75 ± 0.02 and a specificity of 0.74 ± 0.07. The results of our study demonstrate the feasibility of leveraging natural speech for monitoring COPD in real-world settings, offering a promising solution for disease management and even diagnosis.

Using LSTM neural networks for cross‐lingual phonetic speech segmentation with an iterative correction procedure

AbstractThis article describes experiments on speech segmentation using long short‐term memory recurrent neural networks. The main part of the paper deals with multi‐lingual and cross‐lingual segmentation, that is, it is performed on a language different from the one on which the model was trained. The experimental data involves large Czech, English, German, and Russian speech corpora designated for speech synthesis. For optimal multi‐lingual modeling, a compact phonetic alphabet was proposed by sharing and clustering phones of particular languages. Many experiments were performed exploring various experimental conditions and data combinations. We proposed a simple procedure that iteratively adapts the inaccurate default model to the new voice/language. The segmentation accuracy was evaluated by comparison with reference segmentation created by a well‐tuned hidden Markov model‐based framework with additional manual corrections. The resulting segmentation was also employed in a unit selection text‐to‐speech system. The generated speech quality was compared with the reference segmentation by a preference listening test.