Speech Tokens Research Articles

Automated speech analytics in ALS: higher sensitivity of digital articulatory precision over the ALSFRS-R

Objective: Although studies have shown that digital measures of speech detected ALS speech impairment and correlated with the ALSFRS-R speech item, no study has yet compared their performance in detecting speech changes. In this study, we compared the performances of the ALSFRS-R speech item and an algorithmic speech measure in detecting clinically important changes in speech. Importantly, the study was part of a FDA submission which received the breakthrough device designation for monitoring ALS; we provide this paper as a roadmap for validating other speech measures for monitoring disease progression. Methods: We obtained ALSFRS-R speech subscores and speech samples from participants with ALS. We computed the minimum detectable change (MDC) of both measures; using clinician-reported listener effort and a perceptual ratings of severity, we calculated the minimal clinically important difference (MCID) of each measure with respect to both sets of clinical ratings. Results: For articulatory precision, the MDC (.85) was lower than both MCID measures (2.74 and 2.28), and for the ALSFRS-R speech item, MDC (.86) was greater than both MCID measures (.82 and .72), indicating that while the articulatory precision measure detected minimal clinically important differences in speech, the ALSFRS-R speech item did not. Conclusion: The results demonstrate that the digital measure of articulatory precision effectively detects clinically important differences in speech ratings, outperforming the ALSFRS-R speech item. Taken together, the results herein suggest that this speech outcome is a clinically meaningful measure of speech change.

Comparing Speech Production Errors of 3 to 5 Year Old Bengali Children with Cleft Lip and/or Palate in Two Contexts: Word Naming And Sentence Repetition

Research suggests various standardized assessment tools to collect and analyze the speech of the individual with cleft lip and/or palate (CLP). Despite the importance of developing an assessment framework for the growing CLP population, documentation on Bengali cleft speech is limited. This study aims to investigate the phonetic differences that happen in the speech sound production of 3 to 5-year-old Bengali-speaking CLP children in two different contexts: single word and sentence repetition. It also aims to evaluate the effects of these two speech assessment tools in those children’s phonological evaluation to identify the proper evaluation for CLP individuals. In total, 12 Bengali-speaking CLP children from 3 to 5 years old participated in each test. Thus, data has been collected from a sample of 24 with 79-word stimuli. Two types of speech materials were used: single-word naming and sentence repetition tasks. The samples were analyzed using narrow phonetic transcription for targeted speech items. The percentage of consonants correct (PCC), percentage of correct places (PCP), percentage of correct manners (PCM), as well as phonological simplification processes (PSP) were assessed from the samples. Results showed a range of word structures and place-manner features produced by CLP children in both tasks. Also, the PCC, PCP, PCM, and PSP results were considerably less accurate in the sentence repetition task, suggesting that the task may provide a more accurate assessment of the CLP population. Social Science Review, Vol. 40(2), December 2023 Page: 205-224

UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding

The utilization of discrete speech tokens, divided into semantic tokens and acoustic tokens, has been proven superior to traditional acoustic feature mel-spectrograms in terms of naturalness and robustness for text-to-speech (TTS) synthesis. Recent popular models, such as VALL-E and SPEAR-TTS, allow zero-shot speaker adaptation through auto-regressive (AR) continuation of acoustic tokens extracted from a short speech prompt. However, these AR models are restricted to generate speech only in a left-to-right direction, making them unsuitable for speech editing where both preceding and following contexts are provided. Furthermore, these models rely on acoustic tokens, which have audio quality limitations imposed by the performance of audio codec models. In this study, we propose a unified context-aware TTS framework called UniCATS, which is capable of both speech continuation and editing. UniCATS comprises two components, an acoustic model CTX-txt2vec and a vocoder CTX-vec2wav. CTX-txt2vec employs contextual VQ-diffusion to predict semantic tokens from the input text, enabling it to incorporate the semantic context and maintain seamless concatenation with the surrounding context. Following that, CTX-vec2wav utilizes contextual vocoding to convert these semantic tokens into waveforms, taking into consideration the acoustic context. Our experimental results demonstrate that CTX-vec2wav outperforms HifiGAN and AudioLM in terms of speech resynthesis from semantic tokens. Moreover, we show that UniCATS achieves state-of-the-art performance in both speech continuation and editing. Audio samples are available at https://cpdu.github.io/unicats.

Investigation of machine-learning-based stimuli for the remediation of children’s speech errors

Children with speech sound disorders (SSDs) demonstrate difficulty producing phonemes correctly and may exhibit poor speech perception compared to their age-matched peers; however, group differences in speech perception skills remain largely unexplained. Developmental models of speech production posit that children’s ability to discriminate correct and incorrect productions in their own speech may be critical for developing accurate speech production. Historically, when children regularly mispronounce a phoneme, it has been essentially impossible to assess whether they perceive correct versus errors productions in their own speech, thus creating a clinical conundrum. How can we assess a child’s ability to perceive the accuracy of their own phoneme production when they cannot produce a correct production? Recent technological developments allow for acoustic alteration of children’s speech that digitally corrects speech sound errors while preserving natural characteristics of the child’s voice. This machine-learning-based stimuli may then be used as training and feedback tokens for remediation when treating children with speech sound disorders. Such acoustic alteration is possible within an accessible, user-friendly environment that is clinically feasible for speech-language pathologists with little acoustic training. Thus, the purpose of this study is to evaluate the acoustic and perceptual accuracy of acoustically-altered child-speech compared to natural speech tokens.

The Early Subcortical Response at the Fundamental Frequency of Speech Is Temporally Separated from Later Cortical Contributions.

Most parts of speech are voiced, exhibiting a degree of periodicity with a fundamental frequency and many higher harmonics. Some neural populations respond to this temporal fine structure, in particular at the fundamental frequency. This frequency-following response to speech consists of both subcortical and cortical contributions and can be measured through EEG as well as through magnetoencephalography (MEG), although both differ in the aspects of neural activity that they capture: EEG is sensitive to both radial and tangential sources as well as to deep sources, whereas MEG is more restrained to the measurement of tangential and superficial neural activity. EEG responses to continuous speech have shown an early subcortical contribution, at a latency of around 9 msec, in agreement with MEG measurements in response to short speech tokens, whereas MEG responses to continuous speech have not yet revealed such an early component. Here, we analyze MEG responses to long segments of continuous speech. We find an early subcortical response at latencies of 4-11 msec, followed by later right-lateralized cortical activities at delays of 20-58 msec as well as potential subcortical activities. Our results show that the early subcortical component of the FFR to continuous speech can be measured from MEG in populations of participants and that its latency agrees with that measured with EEG. They furthermore show that the early subcortical component is temporally well separated from later cortical contributions, enabling an independent assessment of both components toward further aspects of speech processing.

Аналитический обзор методов автоматического анализа экстралингвистических компонентов спонтанной речи

The accuracy of automatic spontaneous speech recognition systems is far from that of trained speech recognition systems. This is due to the fact that spontaneous speech is not as smooth and failure-free as spontaneous speech. Spontaneous speech varies from speaker to speaker: the quality of phonemes’ pronunciation, the presence of pauses, speech disruptions and extralinguistic items (laughing, coughing, sneezing, and chuckling when expressing emotions of irritation, etc.) interrupt the fluency of verbal speech. However, it is worth noting that extralinguistic items very often carry important paralinguistic information, so it is crucial for automatic spontaneous speech recognition systems not only to identify such phenomena and distinguish them from the verbal components of speech but also to classify them. This review presents an analysis of works on the topic of automatic detection and analysis of extralinguistic items in spontaneous speech. Both individual methods and approaches to the recognition of extralinguistic items in a speech stream, and works related to the multiclass classification of isolatedly recorded extralinguistic units are considered and described. The most popular methods of extralinguistic units’ analysis are neural networks, such as deep neural networks and networks based on transformer models. The basic concepts related to the term extralinguistic items are given, the original systematization of extralinguistic items in the Russian language is proposed, the corpus and databases of audio spoken speech both in Russian and in other languages are described, the data sets of extralinguistic items recorded isolatedly are also given. The accuracy of extralinguistic items recognition increases with the following conditions of work with the speech signal: pre-processing of audio signals of items has shown an increase in the accuracy of separately recorded extralinguistic items classification; consideration of context (analysis of several frames of speech signal) and use of filters for smoothing the time series after extraction of feature vectors showed an increase in accuracy in frame-by-frame analysis of the speech signal with spontaneous speech.

Isolating Neural Signatures of Conscious Speech Perception with a No-Report Sine-Wave Speech Paradigm.

Identifying neural correlates of conscious perception is a fundamental endeavor of cognitive neuroscience. Most studies so far have focused on visual awareness along with trial-by-trial reports of task-relevant stimuli, which can confound neural measures of perceptual awareness with postperceptual processing. Here, we used a three-phase sine-wave speech paradigm that dissociated between conscious speech perception and task relevance while recording EEG in humans of both sexes. Compared with tokens perceived as noise, physically identical sine-wave speech tokens that were perceived as speech elicited a left-lateralized, near-vertex negativity, which we interpret as a phonological version of a perceptual awareness negativity. This response appeared between 200 and 300 ms after token onset and was not present for frequency-flipped control tokens that were never perceived as speech. In contrast, the P3b elicited by task-irrelevant tokens did not significantly differ when the tokens were perceived as speech versus noise and was only enhanced for tokens that were both perceived as speech and relevant to the task. Our results extend the findings from previous studies on visual awareness and speech perception and suggest that correlates of conscious perception, across types of conscious content, are most likely to be found in midlatency negative-going brain responses in content-specific sensory areas.

Universal Speech Token Learning Via Low-Bitrate Neural Codec and Pretrained Representations

Universal Speech Token Learning Via Low-Bitrate Neural Codec and Pretrained Representations

Part of speech-based semantic similarity for RDF Predicate Selection : A benchmarking approach

RDF predicate selection and RDF subject identification are two key steps for effectively processing the natural language queries by the semantic question answering systems. For subject identification, techniques such as Named Entity Recognition are used. For the relatively complex task of predicate selection, Natural Language Processing techniques can be utilized along with knowledge bases to select the semantically matching relationships. Semantic features can be extracted from queries utilizing phrase embeddings of the transformer models. In this paper firstly, a novel Predicate Selection approach on RDF Knowledge Bases has been proposed and implemented utilizing BERT-based phrase embeddings and custom logic for key part-of-speech token combinations. Secondly, the implemented prototype is benchmarked over combinations of key part of speech tokens and stop words for predicate selection accuracy over YAGO Date Facts, demonstrating the effectiveness of our approach. Benchmarking results also highlight the importance of verb stop words for the task of predicate selection.

Contrastive Alveolar/Retroflex Phonemes in Singapore Mandarin Bilinguals: Comprehension Rates for Articulations in Different Accents, and Acoustic Analysis of Productions.

The standard Beijing variety of Mandarin has a clear alveolar-retroflex contrast for phonemes featuring voiceless sibilant frication (i.e., /s/, /ʂ/, /ʈs/, /ʈʂ/, /ʈsʰ/, /ʈʂʰ/). However, some studies show that varieties in the 'outer circle', such in Taiwan, have a reduced contrast for these speech sounds via a process known as 'deretroflexion'. The variety of Mandarin spoken in Singapore is also considered as 'outer circle', as it exhibits influences from Min Nan varieties. We investigated how bilinguals of Singapore Mandarin and English perceive and produce speech tokens in minimal pairs differing only in the alveolar/retroflex place of articulation. In all, 50 participants took part in two tasks. In Task 1, participants performed a lexical identification task for minimal pairs differing only the alveolar/retroflex place of articulation, as spoken by native speakers of two varieties: Beijing Mandarin and Singapore Mandarin. No difference in comprehension of the words was observed between the two varieties indicating that both varieties contain sufficient acoustic information for discrimination. In Task 2, participants read aloud from the list of minimal pairs while their voices were recorded. Acoustic analysis revealed that the phonemes do indeed differ acoustically in terms of center of gravity of the frication and in an alternative measure: long-term averaged spectra. The magnitude of this difference appears to be smaller than previously reported differences for the Beijing variety. These findings show that although some deretroflexion is evident in the speech of bilinguals of the Singaporean variety of Mandarin, it does not translate to ambiguity in the speech signal.

COVID-19 Impacts the Mental Health and Speech Function in Spinocerebellar Ataxia Type 2: Evidences from a Follow-Up Study.

Limited evidence suggests that the SARS-CoV-2 infection can accelerate the progression of neurodegenerative diseases, but this has been not verified in the spinocerebellar ataxias (SCA). The objective of this study is to assess the impact of COVID-19 on the mental health and motor features of SCA2. A follow-up study was carried out in 170 Cuban SCA2 subjects and 87 community controls between 2020 and 2021. All subjects underwent a structured questionnaire to assess the risks of exposure to COVID-19, the confirmation of COVID-19 diagnosis, and the Hospital Anxiety and Depression Scale (HADS). Moreover, 36 subjects underwent the Scale for the Assessment and Rating of ataxia (SARA). The risk of exposure to SARS-CoV-2 and the frequency of COVID-19 were similar between the ataxia cohort and the community controls. Within the ataxia group, significantly increased HADS scores existed at the 2nd visit in both groups, but this increase was more evident for the infected group regarding the depression score. Moreover, a significant within-group increase of SARA score was observed in the infected group but not the non-infected group, which was mainly mediated by the significant increase of the speech item score in the infected group. Similar results were observed within the subgroup of preclinical carriers. Our study identified no selective vulnerability nor protection to COVID-19 in SCA2, but once infected, the patients experienced a deterioration of mental health and speech function, even at preclinical disease stage. These findings set rationales for tele-health approaches that minimize the detrimental effect of COVID-19 on SCA2 progression and identify SCA2 individuals as clinical model to elucidate the link between SARS-CoV-2 infection and neurodegeneration.

Using functional near-infrared spectroscopy in measuring the hearing thresholds in sleeping infants

Functional near-infrared spectroscopy (fNIRS) is a developing technology that uses near-infrared light to image brain activity in the surface layers of the cortexIt measures changes in oxy- haemoglobin (HbO) and deoxy-haemoglobin (HbR) in response to stimuli, which makes it suitable for objectively assessing hearing thresholds by examining the morphology of sound-evoked fNIRS response. This study recruited twelve sleeping infants with no known hearing loss. A natural recording of the /ba/ speech token was used as the stimulus, which was trimmed and concatenated into a 5.4 s stimulus block. The stimulus was presented monaurally between 35 and 80dB SPL using an insert earphone. fNIRS responses were recorded from bilateral pre-frontal and temporal regions. We observed a positive peak at around 5–6 s from stimulus onset and followed by a negative trough at around 10–20 s from stimulus onset. The amplitudes and latencies of this response varied with different stimulus intensity levels. Results showed how features of the fNIRS response changed systematically with intensity level. Characterizing the response differences to different stimulus intensity levels will facilitate the development of a model for assessment of hearing thresholds in this population.

The Impact of Patient Factors on Objective Cochlear Implant Verification Using Acoustic Cortical Auditory-Evoked Potentials

Introduction: Hearing loss is a major global public health issue that negatively impacts quality of life, communication, cognition, social participation, and mental health. The cochlear implant (CI) is the most efficacious treatment for severe-to-profound sensorineural hearing loss. However, variability in outcomes remains high among CI users. Our previous research demonstrated that the existing subjective methodology of CI programming does not consistently produce optimal stimulation for speech perception, thereby limiting the potential for CI users to derive the maximum device benefit to achieve their peak potential. We demonstrated the benefit of utilising the objective method of measuring auditory-evoked cortical responses to speech stimuli as a reliable tool to guide and verify CI programming and, in turn, significantly improve speech perception performance. The present study was designed to investigate the impact of patient- and device-specific factors on the application of acoustically-evoked cortical auditory-evoked potential (aCAEP) measures as an objective clinical tool to verify CI mapping in adult CI users with bilateral deafness (BD). Methods: aCAEP responses were elicited using binaural peripheral auditory stimulation for four speech tokens (/m/, /g/, /t/, and /s/) and recorded by HEARLab™ software in adult BD CI users. Participants were classified into groups according to subjective or objective CI mapping procedures to elicit present aCAEP responses to all four speech tokens. The impact of patient- and device-specific factors on the presence of aCAEP responses and speech perception was investigated between participant groups. Results: Participants were categorised based on the presence or absence of the P1-N1-P2 aCAEP response to speech tokens. Out of the total cohort of adult CI users (n = 132), 63 participants demonstrated present responses pre-optimisation, 37 participants exhibited present responses post-optimisation, and the remaining 32 participants either showed an absent response for at least one speech token post-optimisation or did not accept the optimised CI map adjustments. Overall, no significant correlation was shown between patient and device-specific factors and the presence of aCAEP responses or speech perception scores. Conclusion: This study reinforces that aCAEP measures offer an objective, non-invasive approach to verify CI mapping, irrespective of patient or device factors. These findings further our understanding of the importance of personalised CI rehabilitation through CI mapping to minimise the degree of speech perception variation post-CI and allow all CI users to achieve maximum device benefit.

Online Ternary Classification of Covert Speech by Leveraging the Passive Perception of Speech.

Brain-computer interfaces (BCIs) provide communicative alternatives to those without functional speech. Covert speech (CS)-based BCIs enable communication simply by thinking of words and thus have intuitive appeal. However, an elusive barrier to their clinical translation is the collection of voluminous examples of high-quality CS signals, as iteratively rehearsing words for long durations is mentally fatiguing. Research on CS and speech perception (SP) identifies common spatiotemporal patterns in their respective electroencephalographic (EEG) signals, pointing towards shared encoding mechanisms. The goal of this study was to investigate whether a model that leverages the signal similarities between SP and CS can differentiate speech-related EEG signals online. Ten participants completed a dyadic protocol where in each trial, they listened to a randomly selected word and then subsequently mentally rehearsed the word. In the offline sessions, eight words were presented to participants. For the subsequent online sessions, the two most distinct words (most separable in terms of their EEG signals) were chosen to form a ternary classification problem (two words and rest). The model comprised a functional mapping derived from SP and CS signals of the same speech token (features are extracted via a Riemannian approach). An average ternary online accuracy of 75.3% (60% chance level) was achieved across participants, with individual accuracies as high as 93%. Moreover, we observed that the signal-to-noise ratio (SNR) of CS signals was enhanced by perception-covert modeling according to the level of high-frequency ([Formula: see text]-band) correspondence between CS and SP. These findings may lead to less burdensome data collection for training speech BCIs, which could eventually enhance the rate at which the vocabulary can grow.

Duplex perception reveals brainstem auditory representations are modulated by listeners' ongoing percept for speech.

So-called duplex speech stimuli with perceptually ambiguous spectral cues to one ear and isolated low- versus high-frequency third formant "chirp" to the opposite ear yield a coherent percept supporting their phonetic categorization. Critically, such dichotic sounds are only perceived categorically upon binaural integration. Here, we used frequency-following responses (FFRs), scalp-recorded potentials reflecting phase-locked subcortical activity, to investigate brainstem responses to fused speech percepts and to determine whether FFRs reflect binaurally integrated category-level representations. We recorded FFRs to diotic and dichotic stop-consonants (/da/, /ga/) that either did or did not require binaural fusion to properly label along with perceptually ambiguous sounds without clear phonetic identity. Behaviorally, listeners showed clear categorization of dichotic speech tokens confirming they were heard with a fused, phonetic percept. Neurally, we found FFRs were stronger for categorically perceived speech relative to category-ambiguous tokens but also differentiated phonetic categories for both diotically and dichotically presented speech sounds. Correlations between neural and behavioral data further showed FFR latency predicted the degree to which listeners labeled tokens as "da" versus "ga." The presence of binaurally integrated, category-level information in FFRs suggests human brainstem processing reflects a surprisingly abstract level of the speech code typically circumscribed to much later cortical processing.

Relationships between native and non-native speech perception.

Individuals differ in their ability to perceive and learn unfamiliar speech sounds, but we lack a comprehensive theoretical account that predicts individual differences in this skill. Predominant theories largely attribute difficulties of non-native speech perception to the relationships between non-native speech sounds/contrasts and native-language categories. The goal of the current study was to test whether the predictions made by these theories can be extended to predict individual differences in naive perception of non-native speech sounds or learning of these sounds. Specifically, we hypothesized that the internal structure of native-language speech categories is the cause of difficulty in perception of unfamiliar sounds such that learners who show more graded (i.e., less categorical) perception of sounds in their native language would have an advantage for perceiving non-native speech sounds because they would be less likely to assimilate unfamiliar speech tokens to their native-language categories. We tested this prediction in two experiments in which listeners categorized speech continua in their native language and performed tasks of discrimination or identification of difficult non-native speech sound contrasts. Overall, results did not support the hypothesis that individual differences in categorical perception of native-language speech sounds is responsible for variability in sensitivity to non-native speech sounds. However, participants who responded more consistently on a speech categorization task showed more accurate perception of non-native speech sounds. This suggests that individual differences in non-native speech perception are more related to the stability of phonetic processing abilities than to individual differences in phonetic category structure. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Sensitivity and specificity of the Amyotrophic Lateral Sclerosis Functional Rating Scale-Revised to detect dysarthria in individuals with amyotrophic lateral sclerosis.

Given the widespread use of the Amyotrophic Lateral Sclerosis Functional Rating Scale-Revised (ALSFRS-R) to measure disease progression in ALS and recent reports demonstrating its poor sensitivity, we aimed to determine the sensitivity and specificity of the ALSFRS-R bulbar subscale and speech item to detect validated clinical ratings of dysarthria in individuals with ALS. Paired ALSFRS-R and validated Speech Intelligibility Test (SIT) data from individuals with ALS were analyzed. Trained raters completed duplicate, independent, and blinded ratings of audio recordings to obtain speech intelligibility (%) and speaking rate (words per minute, WPM). Binary dysarthria profiles were derived (dysarthria ≤96% intelligible and/or <150 WPM). Data were obtained using the Kruskal-Wallis test, receiver-operating characteristic (ROC) curve, area under the curve (AUC), sensitivity and specificity percentages, and positive/negative predictive values (PPV/NPV). A total of 250 paired SIT and ALSFRS-R data points were analyzed. Dysarthria was confirmed in 72.4% (n = 181). Dysarthric speakers demonstrated lower ALSFRS-R bulbar subscale (8.9 vs. 11.2) and speech item (2.7 vs. 3.7) scores (P < .0001). The ALSFRS-R bulbar subscale score had an AUC of 0.81 (95% confidence interval [CI] 0.75 to 0.86). A subscale score of ≤11 yielded a sensitivity of 86%, specificity of 57%, PPV of 84%, and NPV of 60% to correctly identify dysarthria status. The ALSFRS-R speech item score demonstrated an AUC of 0.81 to detect dysarthria (95% CI 0.76 to 0.85), with sensitivity of 79%, specificity of 75%, PPV of 89%, and NPV of 58% for a speech item cutpoint of ≤3. The ALSFRS-R bulbar and speech item subscale scores may be useful, inexpensive, and quick tools for monitoring dysarthria status in ALS.

Perception and production of Mandarin-Accented English: The effect of degree of Accentedness on the Interlanguage Speech Intelligibility Benefit for Listeners (ISIB-L) and Talkers (ISIB-T)

Previous research on the Interlanguage Speech Intelligibility Benefit (ISIB) indicates nonnative listeners may have an advantage at understanding nonnative speech of talkers with the same first language (L1) due to shared interlanguage knowledge. The present study offers a comprehensive analysis of various factors that may modulate this advantage, including the proficiency of both the listeners and the talkers, the mapping of phonemes between the L1 and second language (L2), and the acoustic properties of the phones. Accuracy scores on a lexical decision task were used to investigate both native English listeners’ and native Mandarin learners’ of English perception of native English and Mandarin-accented English speech. Results show clear ISIB-L and ISIB-T effects and demonstrate the dynamic nature of ISIB effects, with both being modulated by speaker and listener proficiency. More striking ISIB effects typically occur at the most extreme ends of accentedness. Additionally, an advantage for common-phoneme over unique-phoneme words in nonnative speech was observed. While nonnative productions of common-phoneme words are more accurate than those of unique-phoneme words, for the most accented productions, nonnative listeners are faster to respond to these unique, often mispronounced, productions.The nonnative listener advantage at perceiving nonnative speech depends on various factors, including listener proficiency, speaker proficiency, phoneme characteristics, and the acoustics of specific speech tokens.

Where on the face do we look during phonemic restoration: An eye-tracking study.

Face to face communication typically involves audio and visual components to the speech signal. To examine the effect of task demands on gaze patterns in response to a speaking face, adults participated in two eye-tracking experiments with an audiovisual (articulatory information from the mouth was visible) and a pixelated condition (articulatory information was not visible). Further, task demands were manipulated by having listeners respond in a passive (no response) or an active (button press response) context. The active experiment required participants to discriminate between speech stimuli and was designed to mimic environmental situations which require one to use visual information to disambiguate the speaker's message, simulating different listening conditions in real-world settings. Stimuli included a clear exemplar of the syllable /ba/ and a second exemplar in which the formant initial consonant was reduced creating an /a/-like consonant. Consistent with our hypothesis, results revealed that the greatest fixations to the mouth were present in the audiovisual active experiment and visual articulatory information led to a phonemic restoration effect for the /a/ speech token. In the pixelated condition, participants fixated on the eyes, and discrimination of the deviant token within the active experiment was significantly greater than the audiovisual condition. These results suggest that when required to disambiguate changes in speech, adults may look to the mouth for additional cues to support processing when it is available.

Automatic Multiple Articulator Segmentation in Dynamic Speech MRI Using a Protocol Adaptive Stacked Transfer Learning U-NET Model.

Dynamic magnetic resonance imaging has emerged as a powerful modality for investigating upper-airway function during speech production. Analyzing the changes in the vocal tract airspace, including the position of soft-tissue articulators (e.g., the tongue and velum), enhances our understanding of speech production. The advent of various fast speech MRI protocols based on sparse sampling and constrained reconstruction has led to the creation of dynamic speech MRI datasets on the order of 80-100 image frames/second. In this paper, we propose a stacked transfer learning U-NET model to segment the deforming vocal tract in 2D mid-sagittal slices of dynamic speech MRI. Our approach leverages (a) low- and mid-level features and (b) high-level features. The low- and mid-level features are derived from models pre-trained on labeled open-source brain tumor MR and lung CT datasets, and an in-house airway labeled dataset. The high-level features are derived from labeled protocol-specific MR images. The applicability of our approach to segmenting dynamic datasets is demonstrated in data acquired from three fast speech MRI protocols: Protocol 1: 3 T-based radial acquisition scheme coupled with a non-linear temporal regularizer, where speakers were producing French speech tokens; Protocol 2: 1.5 T-based uniform density spiral acquisition scheme coupled with a temporal finite difference (FD) sparsity regularization, where speakers were producing fluent speech tokens in English, and Protocol 3: 3 T-based variable density spiral acquisition scheme coupled with manifold regularization, where speakers were producing various speech tokens from the International Phonetic Alphabetic (IPA). Segments from our approach were compared to those from an expert human user (a vocologist), and the conventional U-NET model without transfer learning. Segmentations from a second expert human user (a radiologist) were used as ground truth. Evaluations were performed using the quantitative DICE similarity metric, the Hausdorff distance metric, and segmentation count metric. This approach was successfully adapted to different speech MRI protocols with only a handful of protocol-specific images (e.g., of the order of 20 images), and provided accurate segmentations similar to those of an expert human.