Speech Stream Research Articles

Dual-stream Noise and Speech Information Perception based Speech Enhancement

In real-world scenarios, dynamic ambient noise often degrades speech quality, highlighting the need for advanced speech enhancement techniques. Traditional methods, which rely on static embeddings as auxiliary features, struggle to address the complexities of varying noise conditions. To overcome this, we propose a Dual-stream Noise and Speech Information Perception (DNSIP) approach that dynamically detects and processes both noise and speech through innovative information extraction and suppression mechanisms. Initially, non-speech segments predominantly contain environmental noise, while speech segments carry information about the intended speaker. To handle this dynamic nature, real-time voice activity detection (VAD) is employed to accurately differentiate between speech and noise components. Building on VAD estimates, we propose an innovative information extraction framework that selectively extracts relevant noise and speech features from the noisy input, establishing a dual-stream network for concurrent noise and speech learning. To account for the temporal and spectral variability of noise and speech, a frequency-sequence attention mechanism is integrated, enhancing the model’s ability to learn contextual and spectral dependencies. Additionally, an information suppression module is introduced to minimize cross-stream interference by attenuating noise within the speech stream and suppressing speech content within the noise stream. The derived noise and speech spectrograms are then utilized to formulate a minimum mean square error log-spectral amplitude (MMSE-LSA) estimator for robust speech enhancement. Experimental evaluations on the WSJ0 and VCTK+DEMAND datasets demonstrate that our DNSIP approach surpasses existing state-of-the-art methods, underscoring its efficacy in challenging acoustic environments.

Cortical encoding of hierarchical linguistic information when syllabic rhythms are obscured by echoes

In speech perception, low-frequency cortical activity tracks hierarchical linguistic units (e.g., syllables, phrases, and sentences) on top of acoustic features (e.g., speech envelope). Since the fluctuation of speech envelope typically corresponds to the syllabic boundaries, one common interpretation is that the acoustic envelope underlies the extraction of discrete syllables from continuous speech for subsequent linguistic processing. However, it remains unclear whether and how cortical activity encodes linguistic information when the speech envelope does not provide acoustic correlates of syllables. To address the issue, we introduced a frequency-tagging speech stream where the syllabic rhythm was obscured by echoic envelopes and investigated neural encoding of hierarchical linguistic information using electroencephalography (EEG). When listeners attended to the echoic speech, cortical activity showed reliable tracking of syllable, phrase, and sentence levels, among which the higher-level linguistic units elicited more robust neural responses. When attention was diverted from the echoic speech, reliable neural tracking of the syllable level was also observed in contrast to deteriorated neural tracking of the phrase and sentence levels. Further analyses revealed that the envelope aligned with the syllabic rhythm could be recovered from the echoic speech through a neural adaptation model, and the reconstructed envelope yielded higher predictive power for the neural tracking responses than either the original echoic envelope or anechoic envelope. Taken together, these results suggest that neural adaptation and attentional modulation jointly contribute to neural encoding of linguistic information in distorted speech where the syllabic rhythm is obscured by echoes.

End-to-end elevator voice command recognition system based on improved convolutional neural network

Abstract As a common life scenario, the use of voice control in elevator operation can effectively reduce public health risks and improve user experience in the post-epidemic era. However, due to the characteristics of its closure and public space, it is difficult to deploy the elevator. To solve this problem, an end-to-end speech command recognition algorithm based on a new deep neural network is proposed. The algorithm uses the self-built corpus for training, and uses the speech segment interception algorithm to obtain audio segments under the premise of speech stream as input, puts them into the network model in real time for reasoning, and drives the elevator to run according to the output results. The results show that compared with the recognition model using MFCC-CNN, the network has achieved about 10% improvement in accuracy under the premise of smaller computation, and compared with the DTW algorithm, the network has improved the accuracy by about 25%. Finally, a typical deployment environment is constructed to prove the correctness and effectiveness of the method in practical application.

Sensory intelligence for extraction of abstract auditory rules from a speech sound stream in children with cochlear implants

ObjectiveSensory intelligence in the brain helps listeners automatically extract abstract auditory rules formed by invariant acoustic features from complex speech sound streams, presumably serving as the neural basis for speech comprehension. However, whether this intelligence is deficient in children with cochlear implants (CIs) remains unclear. MethodsMandarin Chinese monosyllables shared a flat lexical tone contour to form an abstract auditory rule but differed in other acoustic features to construct a complex speech sound stream. The abstract rule was occasionally violated by monosyllables with a rising or falling lexical tone. ResultsIn normal hearing (NH) children, the abstract auditory rule could be extracted, as revealed by a mismatch negativity (MMN) and a late discriminative negativity (LDN). However, the MMN and LDN were only evoked in CI children with good hearing and speech performance. NH children with a higher speech perception or spatial hearing score had a greater MMN. The LDN was attenuated with increasing age in NH children. ConclusionsThe sensory intelligence for extraction of auditory abstract rules, associated with speech perception, is deficient in CI children. This intelligence may gradually develop during childhood and adolescence. SignificanceDeficient sensory intelligence in CI children may aid in understanding poor speech comprehension in complex environments.

Simulation modeling of the system of managing the psychophysiological state of a person based on non-verbal extralinguistic characteristics

Today, when artificial intelligence systems are being created, the time of constant dia-logue between humans and computer, there is a great demand for intelligent control systems and human-machine systems. These systems are very complex and require analysis and re-search. The publication is devoted to the simulation computer modeling of control of the psy-chological and physical state of a person. The basis of the human-computer dialogue system is the speech-to-text conversion sys-tem. The use of this system is the basis for the creation of computer language control systems. Unlike systems that work on the principle of identifying a single user response to a request and a proposal or a question from a password database, it is necessary to expand the lan-guage interface of the computer. It is necessary for the computer to accept commands from the operator and execute them only if the voice and its timbre match those registered in the database. Such a system will allow you to differentiate access to the computer and ensure se-curity for the user, preserve his privacy or save his personal data. After analyzing the speech activity of the operator, it became clear that it was necessary to develop a system of command control and «speech-to-text» conversion in the fused speech stream. This task is solved by comparing the words of the commands and is part of the problem of automatic recognition and understanding of the operator's human language. On the basis of such systems, a computer language control system can be created. The analysis has shown that unlike text-dependent identification systems, systems with a dialog method implement not only a one-time operator response to a query or question from the password database, but also the extension of its full-fledged language interface. The computer receives commands from the operator and executes them only if the voice matches the one registered in the database. This analysis is effective in the context of real-life speech activity of the operator. The literature review showed that for the development of sys-tems of command control and "speech-to-text" transformation in a stream of fused speech, it is necessary to use "mechanisms" of automatic recognition and understanding of the opera-tor's language.

Can Infants Retain Statistically Segmented Words and Mappings Across a Delay?

Infants are sensitive to statistics in spoken language that aid word-form segmentation and immediate mapping to referents. However, it is not clear whether this sensitivity influences the formation and retention of word-referent mappings across a delay, two real-world challenges that learners must overcome. We tested how the timing of referent training, relative to familiarization with transitional probabilities (TPs) in speech, impacts English-learning 23-month-olds' ability to form and retain word-referent mappings. In Experiment 1, we tested infants' ability to retain TP information across a 10-min delay and use it in the service of word learning. Infants successfully mapped high-TP but not low-TP words to referents. In Experiment 2, infants readily mapped the same words even when they were unfamiliar. In Experiment 3, high- and low-TP word-referent mappings were trained immediately after familiarization, and infants readily remembered these associations 10 min later. In sum, although 23-month-old infants do not need strong statistics to map word forms to referents immediately, or to remember those mappings across a delay, infants are nevertheless sensitive to these statistics in the speech stream, and they influence mapping after a delay. These findings suggest that, by 23 months of age, sensitivity to statistics in speech may impact infants' language development by leading word forms with low coherence to be poorly mapped following even a short period of consolidation.

Effects of absolute pitch on brain activation and functional connectivity during hearing-in-noise perception

Hearing-in-noise (HIN) ability is crucial in speech and music communication. Recent evidence suggests that absolute pitch (AP), the ability to identify isolated musical notes, is associated with HIN benefits. A theoretical account postulates a link between AP ability and neural network indices of segregation. However, how AP ability modulates the brain activation and functional connectivity underlying HIN perception remains unclear. Here we used functional magnetic resonance imaging to contrast brain responses among a sample (n = 45) comprising 15 AP musicians, 15 non-AP musicians, and 15 non-musicians in perceiving Mandarin speech and melody targets under varying signal-to-noise ratios (SNRs: No-Noise, 0, −9 dB). Results reveal that AP musicians exhibited increased activation in auditory and superior frontal regions across both HIN domains (music and speech), irrespective of noise levels. Notably, substantially higher sensorimotor activation was found in AP musicians when the target was music compared to speech. Furthermore, we examined AP effects on neural connectivity using psychophysiological interaction analysis with the auditory cortex as the seed region. AP musicians showed decreased functional connectivity with the sensorimotor and middle frontal gyrus compared to non-AP musicians. Crucially, AP differentially affected connectivity with parietal and frontal brain regions depending on the HIN domain being music or speech. These findings suggest that AP plays a critical role in HIN perception, manifested by increased activation and functional independence between auditory and sensorimotor regions for perceiving music and speech streams.

Attention-Driven Modulation of Auditory Cortex Activity during Selective Listening in a Multispeaker Setting.

Real-world listening settings often consist of multiple concurrent sound streams. To limit perceptual interference during selective listening, the auditory system segregates and filters the relevant sensory input. Previous work provided evidence that the auditory cortex is critically involved in this process and selectively gates attended input toward subsequent processing stages. We studied at which level of auditory cortex processing this filtering of attended information occurs using functional magnetic resonance imaging (fMRI) and a naturalistic selective listening task. Forty-five human listeners (of either sex) attended to one of two continuous speech streams, presented either concurrently or in isolation. Functional data were analyzed using an inter-subject analysis to assess stimulus-specific components of ongoing auditory cortex activity. Our results suggest that stimulus-related activity in the primary auditory cortex and the adjacent planum temporale are hardly affected by attention, whereas brain responses at higher stages of the auditory cortex processing hierarchy become progressively more selective for the attended input. Consistent with these findings, a complementary analysis of stimulus-driven functional connectivity further demonstrated that information on the to-be-ignored speech stream is shared between the primary auditory cortex and the planum temporale but largely fails to reach higher processing stages. Our findings suggest that the neural processing of ignored speech cannot be effectively suppressed at the level of early cortical processing of acoustic features but is gradually attenuated once the competing speech streams are fully segregated.

Are Familiar Objects More Likely to Be Noticed in an Inattentional Blindness Task?

People often fail to notice the presence of unexpected objects when their attention is engaged elsewhere. In dichotic listening tasks, for example, people often fail to notice unexpected content in the ignored speech stream even though they occasionally do notice highly familiar stimuli like their own name (the "cocktail party" effect). Some of the first studies of inattentional blindness were designed as a visual analog of such dichotic listening studies, but relatively few inattentional blindness studies have examined how familiarity affects noticing. We conducted four preregistered inattentional blindness experiments (total N = 1700) to examine whether people are more likely to notice a familiar unexpected object than an unfamiliar one. Experiment 1 replicated evidence for greater noticing of upright schematic faces than inverted or scrambled ones. Experiments 2-4 tested whether participants from different pairs of countries would be more likely to notice their own nation's flag or petrol company logo than those of another country. These experiments repeatedly found little or no evidence that familiarity affects noticing rates for unexpected objects. Frequently encountered and highly familiar stimuli do not appear to overcome inattentional blindness.

Аналитический обзор методов автоматического анализа экстралингвистических компонентов спонтанной речи

The accuracy of automatic spontaneous speech recognition systems is far from that of trained speech recognition systems. This is due to the fact that spontaneous speech is not as smooth and failure-free as spontaneous speech. Spontaneous speech varies from speaker to speaker: the quality of phonemes’ pronunciation, the presence of pauses, speech disruptions and extralinguistic items (laughing, coughing, sneezing, and chuckling when expressing emotions of irritation, etc.) interrupt the fluency of verbal speech. However, it is worth noting that extralinguistic items very often carry important paralinguistic information, so it is crucial for automatic spontaneous speech recognition systems not only to identify such phenomena and distinguish them from the verbal components of speech but also to classify them. This review presents an analysis of works on the topic of automatic detection and analysis of extralinguistic items in spontaneous speech. Both individual methods and approaches to the recognition of extralinguistic items in a speech stream, and works related to the multiclass classification of isolatedly recorded extralinguistic units are considered and described. The most popular methods of extralinguistic units’ analysis are neural networks, such as deep neural networks and networks based on transformer models. The basic concepts related to the term extralinguistic items are given, the original systematization of extralinguistic items in the Russian language is proposed, the corpus and databases of audio spoken speech both in Russian and in other languages are described, the data sets of extralinguistic items recorded isolatedly are also given. The accuracy of extralinguistic items recognition increases with the following conditions of work with the speech signal: pre-processing of audio signals of items has shown an increase in the accuracy of separately recorded extralinguistic items classification; consideration of context (analysis of several frames of speech signal) and use of filters for smoothing the time series after extraction of feature vectors showed an increase in accuracy in frame-by-frame analysis of the speech signal with spontaneous speech.

The Implementation of Automated Speech Recognition (ASR) in ELT Classroom: A Systematic Literature Review from 2012-2023

Automated Speech Recognition (ASR) turns speech audio streams into text. The use of computer-based voice recognition is beneficial for teaching pronunciation. It may also be used to evaluate a learner's speech in a broader context and set up the potential for creating aural interactions between the learner and the computer. The study is aimed to investigate the use of ASR in speaking assessment using a systematic literature review. Automated scores should be considered based on their validity and any potential issues or mistakes with employing technology (ASR) to evaluate speaking. Although some research has been carried out on the use of technology in speaking assessment, there have been few empirical studies using systematic literature reviews to explore more about ASR in education. Therefore, this study is aimed to provide an updated and comprehensive review of the implementation of Automated Speech Recognition (ASR) in education. Ten studies from journals cited in the Taylor Francis, Wiley, and Springer databases were selected. The results show the benefits of ASR in education including students' progress, interaction, and pedagogical contributions. Pedagogical contributions provide collaboration between the human and automated scores. Teachers can use this study to improve speaking assessments using ASR in the classroom.

Rhythmically Modulating Neural Entrainment during Exposure to Regularities Influences Statistical Learning.

The ability to discover regularities in the environment, such as syllable patterns in speech, is known as statistical learning. Previous studies have shown that statistical learning is accompanied by neural entrainment, in which neural activity temporally aligns with repeating patterns over time. However, it is unclear whether these rhythmic neural dynamics play a functional role in statistical learning or whether they largely reflect the downstream consequences of learning, such as the enhanced perception of learned words in speech. To better understand this issue, we manipulated participants' neural entrainment during statistical learning using continuous rhythmic visual stimulation. Participants were exposed to a speech stream of repeating nonsense words while viewing either (1) a visual stimulus with a "congruent" rhythm that aligned with the word structure, (2) a visual stimulus with an incongruent rhythm, or (3) a static visual stimulus. Statistical learning was subsequently measured using both an explicit and implicit test. Participants in the congruent condition showed a significant increase in neural entrainment over auditory regions at the relevant word frequency, over and above effects of passive volume conduction, indicating that visual stimulation successfully altered neural entrainment within relevant neural substrates. Critically, during the subsequent implicit test, participants in the congruent condition showed an enhanced ability to predict upcoming syllables and stronger neural phase synchronization to component words, suggesting that they had gained greater sensitivity to the statistical structure of the speech stream relative to the incongruent and static groups. This learning benefit could not be attributed to strategic processes, as participants were largely unaware of the contingencies between the visual stimulation and embedded words. These results indicate that manipulating neural entrainment during exposure to regularities influences statistical learning outcomes, suggesting that neural entrainment may functionally contribute to statistical learning. Our findings encourage future studies using non-invasive brain stimulation methods to further understand the role of entrainment in statistical learning.

Steganalysis of AMR Speech Stream Based on Multi-Domain Information Fusion

Steganalysis of AMR Speech Stream Based on Multi-Domain Information Fusion

TENet: leveraging transformer encoders for steganalysis of QIM steganography in VoIP speech streams

TENet: leveraging transformer encoders for steganalysis of QIM steganography in VoIP speech streams

Stopy metryczne jako jednostki kształtujące przebieg utworu fortepianowego

Metric Feet as Units Shaping the Course of a Piano Piece: In this paper, I make an attempt to present the analytical possibilities offered by using metric feet as units that shape the course of a (piano) piece. This approach results from cognitive studies that clearly show that the perception of auditory events is based on grouping them into larger units on the basic strong–weak axis. In this approach, the foot is a scheme of one strong sound and one or two weak sounds (in different combinations) that repeats regularly in the course of the work. In this article, I present the most common ways of distinguishing strong sounds: using duration, dynamics, and pitch (features derived analogously from language, distinguishing strong syllables from the speech stream) and characterising the most common feet.

Crosslinguistic differences in initial word recognition

Abstract The present article addresses the extent to which learners’ mental representations, in particular the phonological and lexical representations of learners’ background languages, influence their ability to perceive and extract linguistic units from a novel speech stream. In the study, native speakers of French were exposed to an unfamiliar language, either Polish or Modern Standard Arabic. A word recognition test taken at first exposure revealed important differences in how French speakers/listeners extract words from the Polish or Arabic speech stream, suggesting that source language representations work differently depending on the specificities of the target language. In addition to providing insights into the effects of source and target language properties on speech perception, these results contribute to on-going discussions about what constitutes crosslinguistic influence and conceptual transfer in second language acquisition research.

Statistical word segmentation succeeds given the minimal amount of exposure.

One of the first tasks in language acquisition is word segmentation, a process to extract word forms from continuous speech streams. Statistical approaches to word segmentation have been shown to be a powerful mechanism, in which word boundaries are inferred from sequence statistics. This approach requires the learner to represent the frequency of units from syllable sequences, though accounts differ on how much statistical exposure is required. In this study, we examined the computational limit with which words can be extracted from continuous sequences. First, we discussed why two occurrences of a word in a continuous sequence is the computational lower limit for this word to be statistically defined. Next, we created shortsyllable sequences that contained certain words either two or four times. Learners were presented with these syllable sequences one at a time, immediately followed by a test of the novel words from these sequences. We found that, with the computationally minimal amount of two exposures, words were successfully segmented from continuous sequences. Moreover, longer syllable sequences providing four exposures to words generated more robust learning results. The implications of these results are discussed in terms of how learners segment and store the word candidates from continuous sequences.

Of words and whistles: Statistical learning operates similarly for identical sounds perceived as speech and non-speech

Statistical learning is an ability that allows individuals to effortlessly extract patterns from the environment, such as sound patterns in speech. Some prior evidence suggests that statistical learning operates more robustly for speech compared to non-speech stimuli, supporting the idea that humans are predisposed to learn language. However, any apparent statistical learning advantage for speech could be driven by signal acoustics, rather than the subjective perception per se of sounds as speech. To resolve this issue, the current study assessed whether there is a statistical learning advantage for ambiguous sounds that are subjectively perceived as speech-like compared to the same sounds perceived as non-speech, thereby controlling for acoustic features. We first induced participants to perceive sine-wave speech (SWS)—a degraded form of speech not immediately perceptible as speech—as either speech or non-speech. After this induction phase, participants were exposed to a continuous stream of repeating trisyllabic nonsense words, composed of SWS syllables, and then completed an explicit familiarity rating task and an implicit target detection task to assess learning. Critically, participants showed robust and equivalent performance on both measures, regardless of their subjective speech perception. In contrast, participants who perceived the SWS syllables as more speech-like showed better detection of individual syllables embedded in speech streams. These results suggest that speech perception facilitates processing of individual sounds, but not the ability to extract patterns across sounds. Our findings suggest that statistical learning is not influenced by the perceived linguistic relevance of sounds, and that it may be conceptualized largely as an automatic, stimulus-driven mechanism.

Learners’ cognitive processing problems during comprehension as a basis for L2 listening research

Second language (L2) learners often experience challenges when they listen. This is due to limitations in their ability to recognise words in streams of speech, apply their knowledge of L2 vocabulary and grammar to process the input, and construct an interpretation that is reasonably complete by using their prior and contextual knowledge. In this paper I consider the importance of identifying learners' cognitive processing problems and why it should be a basis or rationale for conducting L2 listening research. I do this by reviewing ideas discussed in my 2000 System paper which reported a study on L2 learners' real-time cognitive problems during perceptual processing, parsing and utilisation, and proposed pedagogical ideas for developing skills and strategies for overcoming these problems. These early findings on learners' real-time listening problems have lent support to the need for investigating how internal factors might impinge on cognitive processes and pedagogical interventions that may help to improve learners' cognitive processing. I offer suggestions for expanding the scope of research on learners' listening difficulties and distilling new questions from some current lines of inquiry. I also recommend that L2 researchers work with researchers in areas such as neuroscience to adopt new tools and methods for investigating learners’ cognitive processes.

Advancements of phonetics in the 21st century: Theoretical and empirical issues of spoken word recognition in phonetic research

How do listeners understand what they are hearing? Humans hearing speech perform spoken word recognition, recognizing what words they are hearing in a speech stream in order to understand the meaning. Phonetics refers to the properties of the speech at a detailed level, particularly below the level of segmental phonemic distinctions. In order to recognize spoken words, listeners have to extract information from the detailed acoustic signal in some way, but theories differ about whether listeners extract phonemes, whole words, or other units, by what mechanism, and they differ on what kinds of information are stored in the lexicon. The process of spoken word recognition can be affected by any number of situations such as the speaker or listener being a non-native of the language or dialect, being a child, having a speech/hearing disability, hearing speech in noise, the speech itself containing variability, or many other situations. Any of these situations can shed light on theoretical questions by giving a fuller picture of how listeners recognize words. This chapter examines what we have learned in these first ∼21 years of the 21st century about how phonetics interacts with spoken word recognition.