Continuous Speech Research Articles

Improved speech intelligibility in the presence of congruent vibrotactile speech input

Vibrotactile stimulation is believed to enhance auditory speech perception, offering potential benefits for cochlear implant (CI) users who may utilize compensatory sensory strategies. Our study advances previous research by directly comparing tactile speech intelligibility enhancements in normal-hearing (NH) and CI participants, using the same paradigm. Moreover, we assessed tactile enhancement considering stimulus non-specific, excitatory effects through an incongruent audio-tactile control condition that did not contain any speech-relevant information. In addition to this incongruent audio-tactile condition, we presented sentences in an auditory only and a congruent audio-tactile condition, with the congruent tactile stimulus providing low-frequency envelope information via a vibrating probe on the index fingertip. The study involved 23 NH listeners and 14 CI users. In both groups, significant tactile enhancements were observed for congruent tactile stimuli (5.3% for NH and 5.4% for CI participants), but not for incongruent tactile stimulation. These findings replicate previously observed tactile enhancement effects. Juxtaposing our study with previous research, the informational content of the tactile stimulus emerges as a modulator of intelligibility: Generally, congruent stimuli enhanced, non-matching tactile stimuli reduced, and neutral stimuli did not change test outcomes. We conclude that the temporal cues provided by congruent vibrotactile stimuli may aid in parsing continuous speech signals into syllables and words, consequently leading to the observed improvements in intelligibility.

The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario.

Seeing the speaker's face greatly improves our speech comprehension in noisy environments. This is due to the brain's ability to combine the auditory and the visual information around us, a process known as multisensory integration. Selective attention also strongly influences what we comprehend in scenarios with multiple speakers-an effect known as the cocktail-party phenomenon. However, the interaction between attention and multisensory integration is not fully understood, especially when it comes to natural, continuous speech. In a recent electroencephalography (EEG) study, we explored this issue and showed that multisensory integration is enhanced when an audiovisual speaker is attended compared to when that speaker is unattended. Here, we extend that work to investigate how this interaction varies depending on a person's gaze behavior, which affects the quality of the visual information they have access to. To do so, we recorded EEG from 31 healthy adults as they performed selective attention tasks in several paradigms involving two concurrently presented audiovisual speakers. We then modeled how the recorded EEG related to the audio speech (envelope) of the presented speakers. Crucially, we compared two classes of model - one that assumed underlying multisensory integration (AV) versus another that assumed two independent unisensory audio and visual processes (A+V). This comparison revealed evidence of strong attentional effects on multisensory integration when participants were looking directly at the face of an audiovisual speaker. This effect was not apparent when the speaker's face was in the peripheral vision of the participants. Overall, our findings suggest a strong influence of attention on multisensory integration when high fidelity visual (articulatory) speech information is available. More generally, this suggests that the interplay between attention and multisensory integration during natural audiovisual speech is dynamic and is adaptable based on the specific task and environment.

Cortical responses time-locked to continuous speech in the high-gamma band depend on selective attention.

Auditory cortical responses to speech obtained by magnetoencephalography (MEG) show robust speech tracking to the speaker's fundamental frequency in the high-gamma band (70-200 Hz), but little is currently known about whether such responses depend on the focus of selective attention. In this study 22 human subjects listened to concurrent, fixed-rate, speech from male and female speakers, and were asked to selectively attend to one speaker at a time, while their neural responses were recorded with MEG. The male speaker's pitch range coincided with the lower range of the high-gamma band, whereas the female speaker's higher pitch range had much less overlap, and only at the upper end of the high-gamma band. Neural responses were analyzed using the temporal response function (TRF) framework. As expected, the responses demonstrate robust speech tracking of the fundamental frequency in the high-gamma band, but only to the male's speech, with a peak latency of ~40 ms. Critically, the response magnitude depends on selective attention: the response to the male speech is significantly greater when male speech is attended than when it is not attended, under acoustically identical conditions. This is a clear demonstration that even very early cortical auditory responses are influenced by top-down, cognitive, neural processing mechanisms.

Validation of the Acoustic Breathiness Index in Speakers of Finnish Language.

Breathiness (perception of turbulence noise in the voice) is one of the major components of hoarseness in dysphonic voices. This study aims to validate a multiparameter analysis tool, the Acoustic Breathiness Index (ABI), for quantification of breathiness in the speaking voice, including both sustained vowels and continuous speech. One hundred and eight speakers with dysphonia (28 M, 80 F, mean age 50, SD 15.4 years) and 87 non-dysphonic controls (18 M, 69 F, mean age 42, SD 14 years) volunteered as participants. They read a standard text and sustained vowel /a:/. Acoustic recordings were made using a head-mounted microphone. Acoustic samples were evaluated perceptually by nine voice experts of different backgrounds (speech therapists, vocologists and laryngologists). Breathiness (B) from the GRBAS scale was rated. Headphones were used in the perceptual analysis. The dysphonic and non-dysphonic speakers differed significantly from each other in the auditory perceptual evaluation of breathiness. A significant difference was also found for ABI, which had a mean value of 2.26 (SD 1.15) for non-dysphonic and 3.07 (SD 1.75) for dysphonic speakers. ABI correlated strongly with B (rs = 0.823, p = 0.01). ABI's power to distinguish the groups was high (88.6%). The highest sensitivity and specificity of ABI (80%) was obtained at threshold value 2.68. ABI is a valid tool for differentiating breathiness in non-dysphonic and dysphonic speakers of Finnish.

A Brief description of Korean Language Phonology and Its History: From the View of Sinosphere

This paper explores the historical and linguistic facets of the Korean language's development, emphasizing its profound ties to the Korean Peninsula and mainland China. Beginning with the use of Han characters dating back to the 12th century BCE, when Jizi sought sanctuary in Korea, the evolution of these characters into the native Korean script, known as "Hangul," in the 15th century is discussed. Hangul's transformation from addressing writing issues to symbolizing Korean cultural and national identity, particularly in the 20th century, is highlighted. The paper also delves into the distinctive vowel characteristics, consonantal complexities, and syllable structures of the Korean language, making it conducive to continuous speech. Additionally, it touches upon the challenges faced by learners, especially those from Chinese-speaking backgrounds, in mastering these linguistic features. This paper serves as a concise overview of the multifaceted nature of the Korean language's historical and structural elements.

Eelbrain, a Python toolkit for time-continuous analysis with temporal response functions.

Even though human experience unfolds continuously in time, it is not strictly linear; instead, it entails cascading processes building hierarchical cognitive structures. For instance, during speech perception, humans transform a continuously varying acoustic signal into phonemes, words, and meaning, and these levels all have distinct but interdependent temporal structures. Time-lagged regression using temporal response functions (TRFs) has recently emerged as a promising tool for disentangling electrophysiological brain responses related to such complex models of perception. Here, we introduce the Eelbrain Python toolkit, which makes this kind of analysis easy and accessible. We demonstrate its use, using continuous speech as a sample paradigm, with a freely available EEG dataset of audiobook listening. A companion GitHub repository provides the complete source code for the analysis, from raw data to group-level statistics. More generally, we advocate a hypothesis-driven approach in which the experimenter specifies a hierarchy of time-continuous representations that are hypothesized to have contributed to brain responses, and uses those as predictor variables for the electrophysiological signal. This is analogous to a multiple regression problem, but with the addition of a time dimension. TRF analysis decomposes the brain signal into distinct responses associated with the different predictor variables by estimating a multivariate TRF (mTRF), quantifying the influence of each predictor on brain responses as a function of time(-lags). This allows asking two questions about the predictor variables: (1) Is there a significant neural representation corresponding to this predictor variable? And if so, (2) what are the temporal characteristics of the neural response associated with it? Thus, different predictor variables can be systematically combined and evaluated to jointly model neural processing at multiple hierarchical levels. We discuss applications of this approach, including the potential for linking algorithmic/representational theories at different cognitive levels to brain responses through computational models with appropriate linking hypotheses.

Frontier Research on Low-Resource Speech Recognition Technology.

With the development of continuous speech recognition technology, users have put forward higher requirements in terms of speech recognition accuracy. Low-resource speech recognition, as a typical speech recognition technology under restricted conditions, has become a research hotspot nowadays because of its low recognition rate and great application value. Under the premise of low-resource speech recognition technology, this paper reviews the research status of feature extraction and acoustic models, and conducts research on resource expansion. Especially in terms of the technical challenges faced by this technology, solutions are proposed, and future research directions are prospected.

Rhythmic modulation of prediction errors: A top-down gating role for the beta-range in speech processing.

Natural speech perception requires processing the ongoing acoustic input while keeping in mind the preceding one and predicting the next. This complex computational problem could be handled by a dynamic multi-timescale hierarchical inferential process that coordinates the information flow up and down the language network hierarchy. Using a predictive coding computational model (Precoss-β) that identifies online individual syllables from continuous speech, we address the advantage of a rhythmic modulation of up and down information flows, and whether beta oscillations could be optimal for this. In the model, and consistent with experimental data, theta and low-gamma neural frequency scales ensure syllable-tracking and phoneme-level speech encoding, respectively, while the beta rhythm is associated with inferential processes. We show that a rhythmic alternation of bottom-up and top-down processing regimes improves syllable recognition, and that optimal efficacy is reached when the alternation of bottom-up and top-down regimes, via oscillating prediction error precisions, is in the beta range (around 20-30 Hz). These results not only demonstrate the advantage of a rhythmic alternation of up- and down-going information, but also that the low-beta range is optimal given sensory analysis at theta and low-gamma scales. While specific to speech processing, the notion of alternating bottom-up and top-down processes with frequency multiplexing might generalize to other cognitive architectures.

Parkinson’s Disease Classification Framework Using Vocal Dynamics in Connected Speech

Parkinson’s disease (PD) classification through speech has been an advancing field of research because of its ease of acquisition and processing. The minimal infrastructure requirements of the system have also made it suitable for telemonitoring applications. Researchers have studied the effects of PD on speech from various perspectives using different speech tasks. Typical speech deficits due to PD include voice monotony (e.g., monopitch), breathy or rough quality, and articulatory errors. In connected speech, these symptoms are more emphatic, which is also the basis for speech assessment in popular rating scales used for PD, like the Unified Parkinson’s Disease Rating Scale (UPDRS) and Hoehn and Yahr (HY). The current study introduces an innovative framework that integrates pitch-synchronous segmentation and an optimized set of features to investigate and analyze continuous speech from both PD patients and healthy controls (HC). Comparison of the proposed framework against existing methods has shown its superiority in classification performance and mitigation of overfitting in machine learning models. A set of optimal classifiers with unbiased decision-making was identified after comparing several machine learning models. The outcomes yielded by the classifiers demonstrate that the framework effectively learns the intrinsic characteristics of PD from connected speech, which can potentially offer valuable assistance in clinical diagnosis.

Audiotactile Stimulation Can Improve Syllable Discrimination through Multisensory Integration in the Theta Frequency Band.

Syllables are an essential building block of speech. We recently showed that tactile stimuli linked to the perceptual centers of syllables in continuous speech can improve speech comprehension. The rate of syllables lies in the theta frequency range, between 4 and 8 Hz, and the behavioral effect appears linked to multisensory integration in this frequency band. Because this neural activity may be oscillatory, we hypothesized that a behavioral effect may also occur not only while but also after this activity has been evoked or entrained through vibrotactile pulses. Here, we show that audiotactile integration regarding the perception of single syllables, both on the neural and on the behavioral level, is consistent with this hypothesis. We first stimulated participants with a series of vibrotactile pulses and then presented them with a syllable in background noise. We show that, at a delay of 200 msec after the last vibrotactile pulse, audiotactile integration still occurred in the theta band and syllable discrimination was enhanced. Moreover, the dependence of both the neural multisensory integration as well as of the behavioral discrimination on the delay of the audio signal with respect to the last tactile pulse was consistent with a damped oscillation. In addition, the multisensory gain is correlated with the syllable discrimination score. Our results therefore evidence the role of the theta band in audiotactile integration and provide evidence that these effects may involve oscillatory activity that still persists after the tactile stimulation.

English Pronunciation Quality Evaluation System Based on Continuous Speech Recognition Technology for Multi-Terminal

This paper presents a novel approach for evaluating the pronunciation quality of English speech using continuous speech recognition technology. The research focuses on the application of artificial intelligence in speech recognition, utilizing web browsers on various terminal devices such as computers, mobile phones, and tablets to allow users to read the provided text aloud. The web program captures audio input from the microphone, records it in MP3 format, and uploads it to the server. The server employs the Whisper model to transcribe the audio into semantic text, which is then compared with the displayed text. By calculating the semantic distance and assessing the accuracy of pronunciation, the system provides an evaluation of pronunciation quality, marking correct and incorrect words. To achieve real-time processing, the compact tiny model is employed, and further optimization is performed using Ctranslate 2, resulting in significant performance improvements.

Statistical word segmentation succeeds given the minimal amount of exposure.

One of the first tasks in language acquisition is word segmentation, a process to extract word forms from continuous speech streams. Statistical approaches to word segmentation have been shown to be a powerful mechanism, in which word boundaries are inferred from sequence statistics. This approach requires the learner to represent the frequency of units from syllable sequences, though accounts differ on how much statistical exposure is required. In this study, we examined the computational limit with which words can be extracted from continuous sequences. First, we discussed why two occurrences of a word in a continuous sequence is the computational lower limit for this word to be statistically defined. Next, we created shortsyllable sequences that contained certain words either two or four times. Learners were presented with these syllable sequences one at a time, immediately followed by a test of the novel words from these sequences. We found that, with the computationally minimal amount of two exposures, words were successfully segmented from continuous sequences. Moreover, longer syllable sequences providing four exposures to words generated more robust learning results. The implications of these results are discussed in terms of how learners segment and store the word candidates from continuous sequences.

Leading and following: Noise differently affects semantic and acoustic processing during naturalistic speech comprehension

Despite the distortion of speech signals caused by unavoidable noise in daily life, our ability to comprehend speech in noisy environments is relatively stable. However, the neural mechanisms underlying reliable speech-in-noise comprehension remain to be elucidated. The present study investigated the neural tracking of acoustic and semantic speech information during noisy naturalistic speech comprehension. Participants listened to narrative audio recordings mixed with spectrally matched stationary noise at three signal-to-ratio (SNR) levels (no noise, 3 dB, -3 dB), and 60-channel electroencephalography (EEG) signals were recorded. A temporal response function (TRF) method was employed to derive event-related-like responses to the continuous speech stream at both the acoustic and the semantic levels. Whereas the amplitude envelope of the naturalistic speech was taken as the acoustic feature, word entropy and word surprisal were extracted via the natural language processing method as two semantic features. Theta-band frontocentral TRF responses to the acoustic feature were observed at around 400 ms following speech fluctuation onset over all three SNR levels, and the response latencies were more delayed with increasing noise. Delta-band frontal TRF responses to the semantic feature of word entropy were observed at around 200 to 600 ms leading to speech fluctuation onset over all three SNR levels. The response latencies became more leading with increasing noise and decreasing speech comprehension and intelligibility. While the following responses to speech acoustics were consistent with previous studies, our study revealed the robustness of leading responses to speech semantics, which suggests a possible predictive mechanism at the semantic level for maintaining reliable speech comprehension in noisy environments.

Attentional Modulation of the Cortical Contribution to the Frequency-Following Response Evoked by Continuous Speech.

Selective attention to one of several competing speakers is required for comprehending a target speaker among other voices and for successful communication with them. It moreover has been found to involve the neural tracking of low-frequency speech rhythms in the auditory cortex. Effects of selective attention have also been found in subcortical neural activities, in particular regarding the frequency-following response related to the fundamental frequency of speech (speech-FFR). Recent investigations have, however, shown that the speech-FFR contains cortical contributions as well. It remains unclear whether these are also modulated by selective attention. Here we used magnetoencephalography to assess the attentional modulation of the cortical contributions to the speech-FFR. We presented both male and female participants with two competing speech signals and analyzed the cortical responses during attentional switching between the two speakers. Our findings revealed robust attentional modulation of the cortical contribution to the speech-FFR: the neural responses were higher when the speaker was attended than when they were ignored. We also found that, regardless of attention, a voice with a lower fundamental frequency elicited a larger cortical contribution to the speech-FFR than a voice with a higher fundamental frequency. Our results show that the attentional modulation of the speech-FFR does not only occur subcortically but extends to the auditory cortex as well.SIGNIFICANCE STATEMENT Understanding speech in noise requires attention to a target speaker. One of the speech features that a listener can use to identify a target voice among others and attend it is the fundamental frequency, together with its higher harmonics. The fundamental frequency arises from the opening and closing of the vocal folds and is tracked by high-frequency neural activity in the auditory brainstem and in the cortex. Previous investigations showed that the subcortical neural tracking is modulated by selective attention. Here we show that attention affects the cortical tracking of the fundamental frequency as well: it is stronger when a particular voice is attended than when it is ignored.

Robustness of lateral tongue bracing in second language speech

Lateral tongue bracing, a universal speech posture, is an actively maintained robust position, akin to standing and locomotion [Gick et al., 2017, JSLHR 60; Liu et al., 2022, JIPA, Phonetica 79]. Studying the robustness of bracing in second-language (L2) speakers can provide insight into learning of speech posture. In this study, we investigate the extent to which speaking in a second language affects the robustness of lateral tongue bracing compared to first language (L1) speech. Participants read two short texts, one in their native language and one in their second language, under a 10 mm bite-block condition. Intra-oral videos are analyzed for the percentage of time bracing occurred in continuous speech. Our preliminary results showed a decrease in the overall percentage of lateral contact during L2 continuous speech compared to L1. These insights invite a reassessment of our existing parallel posture model. We suggest adapting this model to account for the cognitive demands of L2, and the possible role of learning language-specific articulatory settings in postural robustness for L2 speakers.

Silent speech recognition using data augmentation based on a three-dimensional lip model

In this study, we proposed a data augmentation method using a three-dimensional model of a speaker's face for machine lip reading, which estimates the content of speech without using speech data. The proposed method converted a speaker's face into a three-dimensional model using DECA (Detailed Expression Capture and Animation), and generated a large amount of learning data by rotating the three-dimensional model in different directions. Then we obtained the facial features around the lips as time series data using dlib, which is an OpenCV module. The time series data is fed into a recognition model. We introduced end-to-end recognition model which is used in continuous speech recognition. The recognition model is based on DeepSpeech2, and we modified the filter size of CNN (convolutional neural network) and the number of layers of BiGRU (bidirectional gated recurrent unit) to adapt to our goal. We evaluated the proposed method on ten ordinary Japanese words. We used the phoneme error rate as the evaluation index. As a result of the evaluation, the error rate of about 0.32 was achieved even for data where the speaker of the evaluation data was not included in the speaker of the training data.

Subcortical measures of multi-talker speech processing

Communication in everyday acoustic environments is facilitated by the ability to locate, separate, and process speech from different locations. We do this by combining information received by both ears. Typically, the auditory system does not simply relay information delivered by both ears; rather, bilateral information is combined through elaborate binaural pathways in both the brainstem and cortex to allow us to communicate in complex acoustic environments. But traditionally it has been difficult to evaluate the subcortical underpinnings of speech communication using more complex, naturalistic continuous speech stimuli. Recently, the “peaky” speech method was created that uses stories to evoke brainstem responses using EEG. We have recorded brainstem responses from adults with normal hearing who listen to one or multiple narrators, and narrators coming from one or different perceived spatial locations. This talk will highlight the brainstem work undertaken towards creating subcortical measures of speech processing and spatial hearing.

Speaker-induced Suppression in EEG during a Naturalistic Reading and Listening Task.

Speaking elicits a suppressed neural response when compared with listening to others' speech, a phenomenon known as speaker-induced suppression (SIS). Previous research has focused on investigating SIS at constrained levels of linguistic representation, such as the individual phoneme and word level. Here, we present scalp EEG data from a dual speech perception and production task where participants read sentences aloud then listened to playback of themselves reading those sentences. Playback was separated into immediate repetition of the previous trial and randomized repetition of a former trial to investigate if forward modeling of responses during passive listening suppresses the neural response. Concurrent EMG was recorded to control for movement artifact during speech production. In line with previous research, ERP analyses at the sentence level demonstrated suppression of early auditory components of the EEG for production compared with perception. To evaluate whether linguistic abstractions (in the form of phonological feature tuning) are suppressed during speech production alongside lower-level acoustic information, we fit linear encoding models that predicted scalp EEG based on phonological features, EMG activity, and task condition. We found that phonological features were encoded similarly between production and perception. However, this similarity was only observed when controlling for movement by using the EMG response as an additional regressor. Our results suggest that SIS operates at a sensory representational level and is dissociated from higher order cognitive and linguistic processing that takes place during speech perception and production. We also detail some important considerations when analyzing EEG during continuous speech production.

Exploring the effect of sound scene on self-directed hearing-aid gain adjustment

Self-directed gain adjustments may present a plausible solution for fitting a hearing-aid (HA) without the requirement of audiometric tests. In the current experiment, users interacted with their hearing aids using a touchscreen-enabled mobile device while a continuous speech stimulus with background noise was played in the sound field, simulating real-world sound scenes. They explored different settings by dragging a cursor on the touchscreen, experiencing real-time audio changes, and identifying their preferred amplification level. Thirteen older adults with self-identified hearing difficulties participated in the study (RHHI score ≥6). The self-fitting procedure was conducted in two sound scenes: Restaurant and Traffic. The speech level was set at 60 dB-A in both sound scenes, and the signal-to-noise ratios were 5 and 10 dB for the Restaurant and Traffic scenes, respectively. Following the self-fitting procedure, real-ear aided responses were measured for the amplification profiles obtained using the two sound scenes. The real-ear aided responses for the two scenes resembled each other, with a median correlation coefficient across participants of 0.81. Nevertheless, a marginally significant difference was found between them, mostly driven by higher or lower frequency regions. This suggests that scene-specific adjustments of hearing-aid fitting may be beneficial for some users.

Perceptual intake explains variability in statistical word segmentation

One of the first problems in language learning is to segment words from continuous speech. Both prosodic and distributional information can be useful, and it is an important question how the two types of information are integrated. In this paper, we propose that the distinction between input (the statistical properties of the syllable sequence), and intake (how learners perceptually represent the syllable sequence) is a useful framework to integrate different sources of information. We took a novel approach, observing how a large number of syllable sequences were segmented. These sequences had the same transitional probability information for finding word boundaries but different syllables in them. We found large variability in the performance of the segmentation task, suggesting that factors other than the statistical properties of sequences were at play. This variability was explored using the input/intake asymmetry framework, which predicted that factors that shaped the representation of different syllable sequences could explain the variability of learning. We examined two factors, the saliency of the rhythm in these syllable sequences and how familiar the novel word forms in the sequence were to the existing lexicon. Both factors explained the variance in the learnability of different sequences, suggesting that processing of the sequences shaped learning. The implications of these results to computational models of statistical learning and broader implications to language learning were discussed.