Continuous Speech Research Articles

Robustness of lateral tongue bracing in second language speech

Lateral tongue bracing, a universal speech posture, is an actively maintained robust position, akin to standing and locomotion [Gick et al., 2017, JSLHR 60; Liu et al., 2022, JIPA, Phonetica 79]. Studying the robustness of bracing in second-language (L2) speakers can provide insight into learning of speech posture. In this study, we investigate the extent to which speaking in a second language affects the robustness of lateral tongue bracing compared to first language (L1) speech. Participants read two short texts, one in their native language and one in their second language, under a 10 mm bite-block condition. Intra-oral videos are analyzed for the percentage of time bracing occurred in continuous speech. Our preliminary results showed a decrease in the overall percentage of lateral contact during L2 continuous speech compared to L1. These insights invite a reassessment of our existing parallel posture model. We suggest adapting this model to account for the cognitive demands of L2, and the possible role of learning language-specific articulatory settings in postural robustness for L2 speakers.

Silent speech recognition using data augmentation based on a three-dimensional lip model

In this study, we proposed a data augmentation method using a three-dimensional model of a speaker's face for machine lip reading, which estimates the content of speech without using speech data. The proposed method converted a speaker's face into a three-dimensional model using DECA (Detailed Expression Capture and Animation), and generated a large amount of learning data by rotating the three-dimensional model in different directions. Then we obtained the facial features around the lips as time series data using dlib, which is an OpenCV module. The time series data is fed into a recognition model. We introduced end-to-end recognition model which is used in continuous speech recognition. The recognition model is based on DeepSpeech2, and we modified the filter size of CNN (convolutional neural network) and the number of layers of BiGRU (bidirectional gated recurrent unit) to adapt to our goal. We evaluated the proposed method on ten ordinary Japanese words. We used the phoneme error rate as the evaluation index. As a result of the evaluation, the error rate of about 0.32 was achieved even for data where the speaker of the evaluation data was not included in the speaker of the training data.

Subcortical measures of multi-talker speech processing

Communication in everyday acoustic environments is facilitated by the ability to locate, separate, and process speech from different locations. We do this by combining information received by both ears. Typically, the auditory system does not simply relay information delivered by both ears; rather, bilateral information is combined through elaborate binaural pathways in both the brainstem and cortex to allow us to communicate in complex acoustic environments. But traditionally it has been difficult to evaluate the subcortical underpinnings of speech communication using more complex, naturalistic continuous speech stimuli. Recently, the “peaky” speech method was created that uses stories to evoke brainstem responses using EEG. We have recorded brainstem responses from adults with normal hearing who listen to one or multiple narrators, and narrators coming from one or different perceived spatial locations. This talk will highlight the brainstem work undertaken towards creating subcortical measures of speech processing and spatial hearing.

Speaker-induced Suppression in EEG during a Naturalistic Reading and Listening Task.

Speaking elicits a suppressed neural response when compared with listening to others' speech, a phenomenon known as speaker-induced suppression (SIS). Previous research has focused on investigating SIS at constrained levels of linguistic representation, such as the individual phoneme and word level. Here, we present scalp EEG data from a dual speech perception and production task where participants read sentences aloud then listened to playback of themselves reading those sentences. Playback was separated into immediate repetition of the previous trial and randomized repetition of a former trial to investigate if forward modeling of responses during passive listening suppresses the neural response. Concurrent EMG was recorded to control for movement artifact during speech production. In line with previous research, ERP analyses at the sentence level demonstrated suppression of early auditory components of the EEG for production compared with perception. To evaluate whether linguistic abstractions (in the form of phonological feature tuning) are suppressed during speech production alongside lower-level acoustic information, we fit linear encoding models that predicted scalp EEG based on phonological features, EMG activity, and task condition. We found that phonological features were encoded similarly between production and perception. However, this similarity was only observed when controlling for movement by using the EMG response as an additional regressor. Our results suggest that SIS operates at a sensory representational level and is dissociated from higher order cognitive and linguistic processing that takes place during speech perception and production. We also detail some important considerations when analyzing EEG during continuous speech production.

Exploring the effect of sound scene on self-directed hearing-aid gain adjustment

Self-directed gain adjustments may present a plausible solution for fitting a hearing-aid (HA) without the requirement of audiometric tests. In the current experiment, users interacted with their hearing aids using a touchscreen-enabled mobile device while a continuous speech stimulus with background noise was played in the sound field, simulating real-world sound scenes. They explored different settings by dragging a cursor on the touchscreen, experiencing real-time audio changes, and identifying their preferred amplification level. Thirteen older adults with self-identified hearing difficulties participated in the study (RHHI score ≥6). The self-fitting procedure was conducted in two sound scenes: Restaurant and Traffic. The speech level was set at 60 dB-A in both sound scenes, and the signal-to-noise ratios were 5 and 10 dB for the Restaurant and Traffic scenes, respectively. Following the self-fitting procedure, real-ear aided responses were measured for the amplification profiles obtained using the two sound scenes. The real-ear aided responses for the two scenes resembled each other, with a median correlation coefficient across participants of 0.81. Nevertheless, a marginally significant difference was found between them, mostly driven by higher or lower frequency regions. This suggests that scene-specific adjustments of hearing-aid fitting may be beneficial for some users.

Perceptual intake explains variability in statistical word segmentation

One of the first problems in language learning is to segment words from continuous speech. Both prosodic and distributional information can be useful, and it is an important question how the two types of information are integrated. In this paper, we propose that the distinction between input (the statistical properties of the syllable sequence), and intake (how learners perceptually represent the syllable sequence) is a useful framework to integrate different sources of information. We took a novel approach, observing how a large number of syllable sequences were segmented. These sequences had the same transitional probability information for finding word boundaries but different syllables in them. We found large variability in the performance of the segmentation task, suggesting that factors other than the statistical properties of sequences were at play. This variability was explored using the input/intake asymmetry framework, which predicted that factors that shaped the representation of different syllable sequences could explain the variability of learning. We examined two factors, the saliency of the rhythm in these syllable sequences and how familiar the novel word forms in the sequence were to the existing lexicon. Both factors explained the variance in the learnability of different sequences, suggesting that processing of the sequences shaped learning. The implications of these results to computational models of statistical learning and broader implications to language learning were discussed.

Examining cortical tracking of the speech envelope in post-stroke aphasia.

People with aphasia have been shown to benefit from rhythmic elements for language production during aphasia rehabilitation. However, it is unknown whether rhythmic processing is associated with such benefits. Cortical tracking of the speech envelope (CTenv) may provide a measure of encoding of speech rhythmic properties and serve as a predictor of candidacy for rhythm-based aphasia interventions. Electroencephalography was used to capture electrophysiological responses while Spanish speakers with aphasia (n = 9) listened to a continuous speech narrative (audiobook). The Temporal Response Function was used to estimate CTenv in the delta (associated with word- and phrase-level properties), theta (syllable-level properties), and alpha bands (attention-related properties). CTenv estimates were used to predict aphasia severity, performance in rhythmic perception and production tasks, and treatment response in a sentence-level rhythm-based intervention. CTenv in delta and theta, but not alpha, predicted aphasia severity. Neither CTenv in delta, alpha, or theta bands predicted performance in rhythmic perception or production tasks. Some evidence supported that CTenv in theta could predict sentence-level learning in aphasia, but alpha and delta did not. CTenv of the syllable-level properties was relatively preserved in individuals with less language impairment. In contrast, higher encoding of word- and phrase-level properties was relatively impaired and was predictive of more severe language impairments. CTenv and treatment response to sentence-level rhythm-based interventions need to be further investigated.

Unsupervised phoneme and word acquisition from continuous speech based on a hierarchical probabilistic generative model

Humans can divide the perceived continuous speech signals, which exhibit double articulation structure, into phonemes and words without explicit boundary points or labels and thus learn a language. In constructive developmental studies, learning the double articulation structure of speech signals is important for realizing robots with human-like language learning abilities. In this study, we propose a novel probabilistic generative model called the Gaussian process-hidden semi-Markov model-based double articulation analyzer (GP-HSMM-DAA), which can learn phonemes and words from continuous speech signals by hierarchically connecting two probabilistic generative models (PGMs), namely, the Gaussian process-hidden semi-Markov model and hidden semi-Markov model. In the proposed model, the parameters of each PGM are mutually and complementarily updated and learned, enabling accurate learning of the phonemes and words. The experimental results reveal that GP-HSMM-DAA can segment continuous speech into phonemes and words with higher accuracy than the conventional method.

An Efficient Voice Authentication Approach Using Improved Deep Convolutional Neural Networks with LSTM Networks

Voice authentication and verification play an essential role in forensic science and security systems in today's technological environment. A challenging issue is making precise voice authentication and verification. This document aims to identify scammers using machine learning techniques to find the best speaker identification and classification combination. Traditionally, researchers have approached voice authentication and speech comparison using separate classification and comparison models. However, the accuracy of training with large amounts of data has made deep learning more popular. In this study, we predict speech recognition in speech datasets using an improved deep convolutional neural network (IDCNN) and long short-term memory (LSTM) network structure. The proposed approach entails three main steps: pretreatment, feature extraction, and classification. We could predict voice authentication and verification in this study with an accuracy of 99.3%. Aside from comparing classification performance, performance metrics for accuracy, sensitivity, and specificity were also computed. The newly introduced methods offer superior error rates, precision, F-score accuracy, and recall compared to conventional algorithms. This research is regarded as a pioneer in the field as a quick and effective way to enhance continuous speech recognition activity.

Double Articulation Analyzer With Prosody for Unsupervised Word and Phone Discovery

Infants acquire words and phonemes from unsegmented speech signals using segmentation cues, such as distributional, prosodic, and co-occurrence cues. Many pre-existing computational models that represent the process tend to focus on distributional or prosodic cues. This paper proposes a nonparametric Bayesian probabilistic generative model called the prosodic hierarchical Dirichlet process-hidden language model (Prosodic HDP-HLM). Prosodic HDP-HLM, an extension of HDP-HLM, considers both prosodic and distributional cues within a single integrative generative model. We conducted three experiments on different types of datasets, and demonstrate the validity of the proposed method. The results show that the Prosodic DAA successfully uses prosodic cues and outperforms a method that solely uses distributional cues. The main contributions of this study are as follows: 1) We develop a probabilistic generative model for time series data including prosody that potentially has a double articulation structure; 2) We propose the Prosodic DAA by deriving the inference procedure for Prosodic HDP-HLM and show that Prosodic DAA can discover words directly from continuous human speech signals using statistical information and prosodic information in an unsupervised manner; 3) We show that prosodic cues contribute to word segmentation more in naturally distributed case words, i.e., they follow Zipf's law.

Semantic processing during continuous speech production: an analysis from eye movements and EEG.

Speech production involves neurological planning and articulatory execution. How speakers prepare for articulation is a significant aspect of speech production research. Previous studies have focused on isolated words or short phrases to explore speech planning mechanisms linked to articulatory behaviors, including investigating the eye-voice span (EVS) during text reading. However, these experimental paradigms lack real-world speech process replication. Additionally, our understanding of the neurological dimension of speech planning remains limited. This study examines speech planning mechanisms during continuous speech production by analyzing behavioral (eye movement and speech) and neurophysiological (EEG) data within a continuous speech production task. The study specifically investigates the influence of semantic consistency on speech planning and the occurrence of "look ahead" behavior. The outcomes reveal the pivotal role of semantic coherence in facilitating fluent speech production. Speakers access lexical representations and phonological information before initiating speech, emphasizing the significance of semantic processing in speech planning. Behaviorally, the EVS decreases progressively during continuous reading of regular sentences, with a slight increase for non-regular sentences. Moreover, eye movement pattern analysis identifies two distinct speech production modes, highlighting the importance of semantic comprehension and prediction in higher-level lexical processing. Neurologically, the dual pathway model of speech production is supported, indicating a dorsal information flow and frontal lobe involvement. The brain network linked to semantic understanding exhibits a negative correlation with semantic coherence, with significant activation during semantic incoherence and suppression in regular sentences. The study's findings enhance comprehension of speech planning mechanisms and offer insights into the role of semantic coherence in continuous speech production. Furthermore, the research methodology establishes a valuable framework for future investigations in this domain.

A Novel Computationally Efficient Approach for Exploring Neural Entrainment to Continuous Speech Stimuli Incorporating Cross-Correlation

A linear system identification technique has been widely used to track neural entrainment in response to continuous speech stimuli. Although the approach of the standard regularization method using ridge regression provides a straightforward solution to estimate and interpret neural responses to continuous speech stimuli, inconsistent results and costly computational processes can arise due to the need for parameter tuning. We developed a novel approach to the system identification method called the detrended cross-correlation function, which aims to map stimulus features to neural responses using the reverse correlation and derivative of convolution. This non-parametric (i.e., no need for parametric tuning) approach can maintain consistent results. Moreover, it provides a computationally efficient training process compared to the conventional method of ridge regression. The detrended cross-correlation function correctly captures the temporal response function to speech envelope and the spectral–temporal receptive field to speech spectrogram in univariate and multivariate forward models, respectively. The suggested model also provides more efficient computation compared to the ridge regression to process electroencephalography (EEG) signals. In conclusion, we suggest that the detrended cross-correlation function can be comparably used to investigate continuous speech- (or sound-) evoked EEG signals.

Speech Recognition Algorithms based Cough Recognition System

This paper introduces an innovative technique for creating a cough detection system that relies on speech recognition algorithms. The strategy utilizes the Kaldi platform, which is open source and incorporates a hybrid system of Gaussian Mixture Model-based Hidden Markov Models (GMM-HMM) through a straightforward monophone training model. Additionally, the study examines the effectiveness of two different feature extraction approaches, Mel Frequency Cepstral Coefficient (MFCC) and Perceptual Linear Prediction (PLP). The proposed system can function as a collection tool for gathering natural and spontaneous cough data from conversations or continuous speech. The paper also compares the Kaldi and CMU Sphinx4 toolkits, concluding that Kaldi’s use of GMM-HMM outperforms CMU Sphinx4.

Evaluating the effects of audiovisual delays on speech understanding with hearables

A key consideration in the development of real-time speech processing algorithms is latency between when speech is first captured and when the processed speech is transmitted. Much of the previous work informing the acceptability of processing latency has focused on the thresholds at which delays are detectable or annoying. This study aimed to explore the effects of delays on responding to a task necessitating speech understanding. Sentence-length audiovisual speech stimuli were modified such that the audio stream was delayed relative to the visual stream as well as to an attenuated copy of the audio stream by 10 and 400 ms, respectively. These stimuli were presented in conjunction with a pictorial referent-choice task to 50 normal-hearing participants. Participants responded slower to the task when the audio stream was delayed by 400 ms relative to the video stream. With an additional attenuated copy of the audio stream synchronized with the video stream, reaction time decreased, but this effect was limited to when the target word appeared near the onset of the sentence, highlighting the complexity of delay perception in the context of continuous speech. Implications for determining permissible processing delays for digital electronic hearing protection devices are discussed.

Age differentially modulates the cortical tracking of the lower and higher level linguistic structures during speech comprehension.

Speech comprehension requires listeners to rapidly parse continuous speech into hierarchically-organized linguistic structures (i.e. syllable, word, phrase, and sentence) and entrain the neural activities to the rhythm of different linguistic levels. Aging is accompanied by changes in speech processing, but it remains unclear how aging affects different levels of linguistic representation. Here, we recorded magnetoencephalography signals in older and younger groups when subjects actively and passively listened to the continuous speech in which hierarchical linguistic structures of word, phrase, and sentence were tagged at 4, 2, and 1Hz, respectively. A newly-developed parameterization algorithm was applied to separate the periodically linguistic tracking from the aperiodic component. We found enhanced lower-level (word-level) tracking, reduced higher-level (phrasal- and sentential-level) tracking, and reduced aperiodic offset in older compared with younger adults. Furthermore, we observed the attentional modulation on the sentential-level tracking being larger for younger than for older ones. Notably, the neuro-behavior analyses showed that subjects' behavioral accuracy was positively correlated with the higher-level linguistic tracking, reversely correlated with the lower-level linguistic tracking. Overall, these results suggest that the enhanced lower-level linguistic tracking, reduced higher-level linguistic tracking and less flexibility of attentional modulation may underpin aging-related decline in speech comprehension.

Does the speaker's eye gaze facilitate infants' word segmentation from continuous speech? An ERP study.

The environment in which infants learn language is multimodal and rich with social cues. Yet, the effects of such cues, such as eye contact, on early speech perception have not been closely examined. This study assessed the role of ostensive speech, signalled through the speaker's eye gaze direction, on infants' word segmentation abilities. A familiarisation-then-test paradigm was used while electroencephalography (EEG) was recorded. Ten-month-old Dutch-learning infants were familiarised with audio-visual stories in which a speaker recited four sentences with one repeated target word. The speaker addressed them either with direct or with averted gaze while speaking. In the test phase following each story, infants heard familiar and novel words presented via audio-only. Infants' familiarity with the words was assessed using event-related potentials (ERPs). As predicted, infants showed a negative-going ERP familiarity effect to the isolated familiarised words relative to the novel words over the left-frontal region of interest during the test phase. While the word familiarity effect did not differ as a function of the speaker's gaze over the left-frontal region of interest, there was also a (not predicted) positive-going early ERP familiarity effect over right fronto-central and central electrodes in the direct gaze condition only. This study provides electrophysiological evidence that infants can segment words from audio-visual speech, regardless of the ostensiveness of the speaker's communication. However, the speaker's gaze direction seems to influence the processing of familiar words. RESEARCH HIGHLIGHTS: We examined 10-month-old infants' ERP word familiarity response using audio-visual stories, in which a speaker addressed infants with direct or averted gaze while speaking. Ten-month-old infants can segment and recognise familiar words from audio-visual speech, indicated by their negative-going ERP response to familiar, relative to novel, words. This negative-going ERP word familiarity effect was present for isolated words over left-frontal electrodes regardless of whether the speaker offered eye contact while speaking. An additional positivity in response to familiar words was observed for direct gaze only, over right fronto-central and central electrodes.

Validation of the Acoustic Voice Quality Index and the Acoustic Breathiness Index in the Persian Language

This study aimed to validate the acoustic voice quality index version 03.01 (AVQIv3) and the acoustic breathiness index (ABI) in the Persian language. Six judges assessed the severity degree of the abnormal overall voice quality (hoarseness) and breathiness from continuous speech (cs) and sustained vowel (sv) in 136 voices samples of dysphonic and 45 vocally healthy participants. Firstly, the time length of cs and sv was balanced out for a higher level of ecological validity. Secondly, the concurrent validity and diagnostic accuracy were statistically analyzed with the Spearman rank-order correlation, and the receiver operating characteristics, likelihood ratio, and Youden index. There was substantial inter-rater reliability between judges regarding hoarseness and breathiness. Twenty syllables were identified as the standardized number of syllables for the cs part. Sufficient correlations were found between AVQI and hoarseness (rs =0.74), and between ABI and breathiness (rs =0.74), respectively. The thresholds of 1.63 (sensitivity of 73.2% and specificity of 91%) and 2.97 (sensitivity of 70% and specificity of 87%) yielded the highest level of diagnostic findings for AVQI and ABI, respectively. The AVQIv3 and ABI are reliable indices to quantify the abnormal overall voice quality and breathiness in the Persian language.

Decoding articulatory and phonetic components of naturalistic continuous speech from the distributed language network

Objective. The speech production network relies on a widely distributed brain network. However, research and development of speech brain–computer interfaces (speech-BCIs) has typically focused on decoding speech only from superficial subregions readily accessible by subdural grid arrays—typically placed over the sensorimotor cortex. Alternatively, the technique of stereo-electroencephalography (sEEG) enables access to distributed brain regions using multiple depth electrodes with lower surgical risks, especially in patients with brain injuries resulting in aphasia and other speech disorders. Approach. To investigate the decoding potential of widespread electrode coverage in multiple cortical sites, we used a naturalistic continuous speech production task. We obtained neural recordings using sEEG from eight participants while they read aloud sentences. We trained linear classifiers to decode distinct speech components (articulatory components and phonemes) solely based on broadband gamma activity and evaluated the decoding performance using nested five-fold cross-validation. Main Results. We achieved an average classification accuracy of 18.7% across 9 places of articulation (e.g. bilabials, palatals), 26.5% across 5 manner of articulation (MOA) labels (e.g. affricates, fricatives), and 4.81% across 38 phonemes. The highest classification accuracies achieved with a single large dataset were 26.3% for place of articulation, 35.7% for MOA, and 9.88% for phonemes. Electrodes that contributed high decoding power were distributed across multiple sulcal and gyral sites in both dominant and non-dominant hemispheres, including ventral sensorimotor, inferior frontal, superior temporal, and fusiform cortices. Rather than finding a distinct cortical locus for each speech component, we observed neural correlates of both articulatory and phonetic components in multiple hubs of a widespread language production network. Significance. These results reveal the distributed cortical representations whose activity can enable decoding speech components during continuous speech through the use of this minimally invasive recording method, elucidating language neurobiology and neural targets for future speech-BCIs.

Robust neural tracking of linguistic speech representations using a convolutional neural network

Objective. When listening to continuous speech, populations of neurons in the brain track different features of the signal. Neural tracking can be measured by relating the electroencephalography (EEG) and the speech signal. Recent studies have shown a significant contribution of linguistic features over acoustic neural tracking using linear models. However, linear models cannot model the nonlinear dynamics of the brain. To overcome this, we use a convolutional neural network (CNN) that relates EEG to linguistic features using phoneme or word onsets as a control and has the capacity to model non-linear relations. Approach. We integrate phoneme- and word-based linguistic features (phoneme surprisal, cohort entropy (CE), word surprisal (WS) and word frequency (WF)) in our nonlinear CNN model and investigate if they carry additional information on top of lexical features (phoneme and word onsets). We then compare the performance of our nonlinear CNN with that of a linear encoder and a linearized CNN. Main results. For the non-linear CNN, we found a significant contribution of CE over phoneme onsets and of WS and WF over word onsets. Moreover, the non-linear CNN outperformed the linear baselines. Significance. Measuring coding of linguistic features in the brain is important for auditory neuroscience research and applications that involve objectively measuring speech understanding. With linear models, this is measurable, but the effects are very small. The proposed non-linear CNN model yields larger differences between linguistic and lexical models and, therefore, could show effects that would otherwise be unmeasurable and may, in the future, lead to improved within-subject measures and shorter recordings.

Speech Segmentation and Cross-Situational Word Learning in Parallel.

Language learners track conditional probabilities to find words in continuous speech and to map words and objects across ambiguous contexts. It remains unclear, however, whether learners can leverage the structure of the linguistic input to do both tasks at the same time. To explore this question, we combined speech segmentation and cross-situational word learning into a single task. In Experiment 1, when adults (N = 60) simultaneously segmented continuous speech and mapped the newly segmented words to objects, they demonstrated better performance than when either task was performed alone. However, when the speech stream had conflicting statistics, participants were able to correctly map words to objects, but were at chance level on speech segmentation. In Experiment 2, we used a more sensitive speech segmentation measure to find that adults (N = 35), exposed to the same conflicting speech stream, correctly identified non-words as such, but were still unable to discriminate between words and part-words. Again, mapping was above chance. Our study suggests that learners can track multiple sources of statistical information to find and map words to objects in noisy environments. It also prompts questions on how to effectively measure the knowledge arising from these learning experiences.