Speech Learning Research Articles

Employing deep learning model to evaluate speech information in acoustic simulations of Cochlear implants

Acoustic vocoders play a key role in simulating the speech information available to cochlear implant (CI) users. Traditionally, the intelligibility of vocoder CI simulations is assessed through speech recognition experiments with normally-hearing subjects, a process that can be time-consuming, costly, and subject to individual variability. As an alternative approach, we utilized an advanced deep learning speech recognition model to investigate the intelligibility of CI simulations. We evaluated model’s performance on vocoder-processed words and sentences with varying vocoder parameters. The number of vocoder bands, frequency range, and envelope dynamic range were adjusted to simulate sound processing settings in CI devices. Additionally, we manipulated the low-cutoff frequency and intensity quantization of vocoder envelopes to simulate psychophysical temporal and intensity resolutions in CI patients. The results were evaluated within the context of the audio analysis performed in the model. Interestingly, the deep learning model, despite not being originally designed to mimic human speech processing, exhibited a human-like response to alterations in vocoder parameters, resembling existing human subject results. This approach offers significant time and cost savings compared to testing human subjects, and eliminates learning and fatigue effects during testing. Our findings demonstrate the potential of speech recognition models in facilitating auditory research.

Deep learning for speech denoising with improved Wiener approach

Deep learning for speech denoising with improved Wiener approach

Dual-stream Noise and Speech Information Perception based Speech Enhancement

In real-world scenarios, dynamic ambient noise often degrades speech quality, highlighting the need for advanced speech enhancement techniques. Traditional methods, which rely on static embeddings as auxiliary features, struggle to address the complexities of varying noise conditions. To overcome this, we propose a Dual-stream Noise and Speech Information Perception (DNSIP) approach that dynamically detects and processes both noise and speech through innovative information extraction and suppression mechanisms. Initially, non-speech segments predominantly contain environmental noise, while speech segments carry information about the intended speaker. To handle this dynamic nature, real-time voice activity detection (VAD) is employed to accurately differentiate between speech and noise components. Building on VAD estimates, we propose an innovative information extraction framework that selectively extracts relevant noise and speech features from the noisy input, establishing a dual-stream network for concurrent noise and speech learning. To account for the temporal and spectral variability of noise and speech, a frequency-sequence attention mechanism is integrated, enhancing the model’s ability to learn contextual and spectral dependencies. Additionally, an information suppression module is introduced to minimize cross-stream interference by attenuating noise within the speech stream and suppressing speech content within the noise stream. The derived noise and speech spectrograms are then utilized to formulate a minimum mean square error log-spectral amplitude (MMSE-LSA) estimator for robust speech enhancement. Experimental evaluations on the WSJ0 and VCTK+DEMAND datasets demonstrate that our DNSIP approach surpasses existing state-of-the-art methods, underscoring its efficacy in challenging acoustic environments.

Языковое образование онлайн: принципы создания размеченного корпуса ошибок в специализированном английском языке русскоязычных работников IT-сферы

Today, academic corpora are a matter of great interest to many linguists. The article examines the importance of academic corpora in modern linguistics and pedagogy as a tool for identifying typical errors in the speech of foreign language students. They reveal problems in foreign language acquisition and create new teaching methods. The paper reviews the existing academic corpora and error classifications. The authors developed a corpus of oral speech errors typical of Russian-speaking IT students that study English for IT purposes. The corpus relied on 50 video recordings of in-class activities in which students communicated with English-speaking IT specialists and made up dialogues in English. The corpus made it possible to identify the main difficulties experienced by adult learners of English for IT purposes. It involves a tagged classification of typical errors divided by language level into morphological, syntactic, lexical, and phonetic ones. The research demonstrated good prospects for developing other academic speech corpora with automated error tagging based on the speech of English learners.

The ecological nature of the classroom environment for the perceptual learning of Additional Language speech

The ecological nature of the classroom environment for the perceptual learning of Additional Language speech

Decode, Move and Speak! Self-supervised Learning of Speech Units, Gestures, and Sound Relationships Using Vocal Imitation

Abstract Speech learning encompasses mastering a complex motor system to produce speech sounds from articulatory gestures while simultaneously uncovering discrete units that provide entry to the linguistic system. Remarkably, children acquire these associations between speech sounds, articulatory gestures, and linguistic units in a weakly supervised manner, without the need for explicit labeling of auditory inputs or access to target articulatory gestures. This study uses self-supervised deep learning to investigate the respective roles of sounds, gestures, and linguistic units in speech acquisition and control. In a first experiment, we analyzed the quantized representations learned by vector-quantized variational autoencoders (VQ-VAE) from ground truth acoustic and articulatory data using ABX tests. We show an interesting complementarity between acoustic and articulatory modalities that may help in the discovery of phonemes. In a second experiment, we introduce a computational agent that repeats auditory speech inputs by controlling a virtual vocal apparatus. This agent integrates an articulatory synthesizer capable of reproducing diverse speech stimuli from interpretable parameters, along with two internal models implementing the articulatory-to-acoustic (forward) and acoustic-to-articulatory (inverse) mapping, respectively. Additionally, two inductive biases are used to regularize the ill-posed acoustic-to-articulatory inverse mapping. In line with the first experiment, we explore the complementarity between the auditory input and the articulatory parameters inferred by the agent. We also evaluate the impact of discretizing auditory inputs using VQ-VAE. While the majority of the agent’s productions are intelligible (according to perceptual evaluations), our analysis highlights inconsistencies in the underlying articulatory trajectories. In particular, we show that the agent’s productions only partially reproduce the complementarity between the auditory and articulatory modalities observed in humans.

Park or bark? VOT production of English bilabial stops in Omani Arabic speakers of English

This study compares Voice Onset Time (VOT) patterns of word-initial English/b/and/p/in three groups: Omani undergraduate students, Omani professors and native English speakers (NES). It aims to investigate whether Omani Arabic students and professors can produce native-like VOTs of the English bilabial stops compared to NESs, and to what extent the findings support speech learning theories, such as the Speech Learning Model and the Contrastive Analysis Hypothesis.Words containing/b, p/in two different phonetic environments (CV and CC) were elicited from the participants using a reading task. Also, Arabic words containing word-initial/b/were elicited from Omani professors to compare the VOT of Arabic and English/b/. The results revealed a significant difference between the mean VOT of Arabic and English/b/. A significant difference in the VOT patterns of/p/between students and NESs were also found, but not between professors and NESs. Students'/p/productions were characterized with shorter lag compared to those of professors or NESs. Variability in the production of English/b/existed among Omani speakers. While VOT production in NESs were characterized with short lag, students' and professors’ productions were characterized sometimes with voicing lead and other times with short/long lag. VOT values of/b/and/p/were not significantly affected by the phonetic context. The findings are discussed in relation to the Speech Learning Model and the Contrastive Analysis Hypothesis.

The relationship of speech perception and speech production: It's complicated.

A widely held belief is that speech perception and speech production are tightly linked, with each modality available to help with learning in the other modality. This positive relationship is often summarized as perception and production being "two sides of the same coin." There are, indeed, many situations that have shown this mutually supportive relationship. However, there is a growing body of research showing very different results, with the modalities operating independently, or even in opposition to each other. We review the now-sizeable literature demonstrating the negative effect that speech production can have on perceptual learning of speech, at multiple levels (particularly at the lexical and sublexical levels). By comparing the situations that show this pattern with ones in which more positive interactions occur, we provide an initial account of why the different outcomes are found, identifying factors that lead to either positive or negative effects of production on perception. The review clarifies the complex relationship that exists between the two modalities: They are indeed linked, but their relationship is more complicated than is suggested by the notion that they are two sides of the same coin.

Deep Learning for Visual Speech Analysis: A Survey.

Visual speech, referring to the visual domain of speech, has attracted increasing attention due to its wide applications, such as public security, medical treatment, military defense, and film entertainment. As a powerful AI strategy, deep learning techniques have extensively promoted the development of visual speech learning. Over the past five years, numerous deep learning based methods have been proposed to address various problems in this area, especially automatic visual speech recognition and generation. To push forward future research on visual speech, this paper will present a comprehensive review of recent progress in deep learning methods on visual speech analysis. We cover different aspects of visual speech, including fundamental problems, challenges, benchmark datasets, a taxonomy of existing methods, and state-of-the-art performance. Besides, we also identify gaps in current research and discuss inspiring future research directions.

Analyzing Wav2Vec 1.0 Embeddings for Cross-Database Parkinson's Disease Detection and Speech Features Extraction.

Advancements in deep learning speech representations have facilitated the effective use of extensive unlabeled speech datasets for Parkinson's disease (PD) modeling with minimal annotated data. This study employs the non-fine-tuned wav2vec 1.0 architecture to develop machine learning models for PD speech diagnosis tasks, such as cross-database classification and regression to predict demographic and articulation characteristics. The primary aim is to analyze overlapping components within the embeddings on both classification and regression tasks, investigating whether latent speech representations in PD are shared across models, particularly for related tasks. Firstly, evaluation using three multi-language PD datasets showed that wav2vec accurately detected PD based on speech, outperforming feature extraction using mel-frequency cepstral coefficients in the proposed cross-database classification scenarios. In cross-database scenarios using Italian and English-read texts, wav2vec demonstrated performance comparable to intra-dataset evaluations. We also compared our cross-database findings against those of other related studies. Secondly, wav2vec proved effective in regression, modeling various quantitative speech characteristics related to articulation and aging. Ultimately, subsequent analysis of important features examined the presence of significant overlaps between classification and regression models. The feature importance experiments discovered shared features across trained models, with increased sharing for related tasks, further suggesting that wav2vec contributes to improved generalizability. The study proposes wav2vec embeddings as a next promising step toward a speech-based universal model to assist in the evaluation of PD.

Mengenali dan Menangani Speech Delay Pada Anak

The language development problems, especially speech delay, are developmental problems that are often found. Research in several countries states that about 6 to 8% of preschool children have speech delay problems (Boyle, Gillham, & Smith, 1996; Tomblin, Smith, & Zhang, 1997 in Law et all., also Macias and Wagner, 2005). The purpose of this study is to recognize and treat speech delay in children. This research method is a literature study. The result of this study is an explanation of children's speech development, speech problems in children, the definition, characteristics and causes of speech delay in children and how to deal with speech delay in children. This study concludes that speech learning involves children's receptive and expressive speech. Problems in speech development can also be seen and addressed from these two bases.

Deep learning model of semantic direction exploration based on English V+able corpus distribution and semantic roles

In order to improve English learning efficiency, this paper constructs a deep learning model of semantic orientation exploration based on English V+able corpus distribution and semantic roles. This article combines the practical needs of English learning and establishes an ILP model with the optimization objective of minimizing spectrum resource occupation. A traffic grooming based time aware multipath RSA algorithm (HMRSA-TG) is proposed to solve the standardization problem of English speech recognition. To improve the system efficiency of intelligent English learning systems, a traffic grooming based time aware multipath RSA algorithm (HMRSA-TG) is proposed. Through research, it has been verified that the semantic orientation exploration deep learning model based on the distribution of semantic roles in English V+able corpora can effectively improve the effectiveness of English speech learning. The corpus model proposed in this paper can provide a reliable benchmark database for many speech problem learners and play an important role in English translation software in recognizing input speech with different accents

Enhancing Second Language Speech: The Roles Of Task Repetition And Planning In Improving L2 English Production

This paper explores the impact of task repetition and planning conditions on the complexity, accuracy, and fluency (CAF) of second language (L2) speech production. By conducting a case study on an L2 English learner, the study investigates how the learner's speech performance is characterized under the CAF construct and how it is affected by task repetition, the absence of pre-task planning, and the presence of unpressured within-task planning. The study employs various measures to assess lexical and grammatical complexity, accuracy, and fluency. The findings suggest that while the lack of pre-task planning did not completely compromise complexity, accuracy remained low despite the unpressured within-task planning condition, and fluency was negatively impacted. However, task repetition led to improvements in all three dimensions of CAF. The paper concludes with pedagogical implications for L2 instruction, highlighting the importance of strategic and guided pre-task planning, multiple-task repetitions, and learner self-evaluation to enhance CAF in L2 speech production.

Individual differences in visual pattern completion predict adaptation to degraded speech

Recognizing acoustically degraded speech relies on predictive processing whereby incomplete auditory cues are mapped to stored linguistic representations via pattern recognition processes. While listeners vary in their ability to recognize degraded speech, performance improves when a written transcription is presented, allowing completion of the partial sensory pattern to preexisting representations. Building on work characterizing predictive processing as pattern completion, we examined the relationship between domain-general pattern recognition and individual variation in degraded speech learning. Participants completed a visual pattern recognition task to measure individual-level tendency towards pattern completion. Participants were also trained to recognize noise-vocoded speech with written transcriptions and tested on speech recognition pre- and post-training using a retrieval-based transcription task. Listeners significantly improved in recognizing speech after training, and pattern completion on the visual task predicted improvement for novel items. The results implicate pattern completion as a domain-general learning mechanism that can facilitate speech adaptation in challenging contexts.

A Review of the Influence of Wu Dialect on English Pronunciation Acquisition

The Wu dialect is widely spoken in Wu-speaking regions of China, which covers a considerable number of English learners. However, the phonetic system of the Wu dialect differs significantly from that of English, which greatly affects the English learning process of Wu-speaking learners, thus attracting numerous researchers to conduct related studies. This paper provides a comprehensive overview of research on the influence of Wu dialect on English phonological learning over the past two decades and presents a selection of twenty-seven papers to explore the theoretical framework, the research methodology, and the specific impacts encountered by English language learners who speak Wu dialect. Among them, the theoretical discussion centered on language transfer theory, Fleges Speech Learning Model and Bests Perceptual Assimilation Model; the research methodology consisted of qualitative analysis and empirical study; and the focus was placed on positive and negative transfer effects, i.e., the challenges posed by the absence or presence of phonological differences in the Wu dialect to the learners of English. The findings reveal significant segmental and suprasegmental influences in English pronunciation acquisition, with segmental challenges including differentiation of consonants and incomplete articulation of vowels, and suprasegmental challenges including stress placement, intonation patterns, and phonological rhythm. In addition, the discussion highlights the need for more empirical studies to supplement qualitative analyses, the consideration of regional variations within Wu dialect, and the exploration of teaching strategies informed by research findings. Therefore, the paper advocates for future research focusing on regional differences, deeper exploration of suprasegmental influences, integration of research into teaching practices, and diversified research methods.

Auditory-motor entrainment and listening experience shape the perceptual learning of concurrent speech.

Plasticity from auditory experience shapes the brain's encoding and perception of sound. Though prior research demonstrates that neural entrainment (i.e., brain-to-acoustic synchronization) aids speech perception, how long- and short-term plasticity influence entrainment to concurrent speech has not been investigated. Here, we explored neural entrainment mechanisms and the interplay between short- and long-term neuroplasticity for rapid auditory perceptual learning of concurrent speech sounds in young, normal-hearing musicians and nonmusicians. Participants learned to identify double-vowel mixtures during ∼45 min training sessions with concurrent high-density EEG recordings. We examined the degree to which brain responses entrained to the speech-stimulus train (∼9 Hz) to investigate whether entrainment to speech prior to behavioral decision predicted task performance. Source and directed functional connectivity analyses of the EEG probed whether behavior was driven by group differences auditory-motor coupling. Both musicians and nonmusicians showed rapid perceptual learning in accuracy with training. Interestingly, listeners' neural entrainment strength prior to target speech mixtures predicted behavioral identification performance; stronger neural synchronization was observed preceding incorrect compared to correct trial responses. We also found stark hemispheric biases in auditory-motor coupling during speech entrainment, with greater auditory-motor connectivity in the right compared to left hemisphere for musicians (R>L) but not in nonmusicians (R=L). Our findings confirm stronger neuroacoustic synchronization and auditory-motor coupling during speech processing in musicians. Stronger neural entrainment to rapid stimulus trains preceding incorrect behavioral responses supports the notion that alpha-band (∼10 Hz) arousal/suppression in brain activity is an important modulator of trial-by-trial success in perceptual processing.

Adaptive multi-task learning for speech to text translation

End-to-end speech to text translation aims to directly translate speech from one language into text in another, posing a challenging cross-modal task particularly in scenarios of limited data. Multi-task learning serves as an effective strategy for knowledge sharing between speech translation and machine translation, which allows models to leverage extensive machine translation data to learn the mapping between source and target languages, thereby improving the performance of speech translation. However, in multi-task learning, finding a set of weights that balances various tasks is challenging and computationally expensive. We proposed an adaptive multi-task learning method to dynamically adjust multi-task weights based on the proportional losses incurred during training, enabling adaptive balance in multi-task learning for speech to text translation. Moreover, inherent representation disparities across different modalities impede speech translation models from harnessing textual data effectively. To bridge the gap across different modalities, we proposed to apply optimal transport in the input of end-to-end model to find the alignment between speech and text sequences and learn the shared representations between them. Experimental results show that our method effectively improved the performance on the Tibetan-Chinese, English-German, and English-French speech translation datasets.

Research on the promotion of intelligent entertainment voice robots in personalized English learning based on data mining and gamified teaching experience

With the development of educational technology and the arrival of the intelligent era, intelligent entertainment voice robots have gradually become a research hotspot in the field of education. This study aims to explore the application of intelligent entertainment voice robots based on data mining in the promotion of personalized English learning, and combine with gamified virtual teaching to improve learners’ learning interest and learning effect. The study uses data mining technology to analyze learners’ learning data, interest preferences and learning performance, and provides learners with targeted learning resources and gamified virtual teaching environment based on personalized recommendation. The study creates a virtual teaching environment presented in the form of games, which contains various interesting tasks, challenges and reward mechanisms. Learners can participate in the virtual teaching environment at the same time, experience the fun and stimulation of games, so as to stimulate the interest and motivation of learning. Learners can engage in interactive speech learning through conversations with intelligent entertainment speech robots. The robot can provide real-time answers to learners’ questions, provide personalized learning suggestions, and provide encouragement and feedback. By interacting with robots, learners can get more flexible and personalized learning support, improving learning effectiveness and satisfaction. Through experiments and data analysis, it is found that intelligent entertainment voice robot based on data mining and gamification teaching experience can effectively improve learners’ learning interest and learning effect. Learners show higher engagement and motivation in personalized recommended learning resources and gamified virtual teaching environments.

Experience of intelligent speech robot in music online classroom based on deep learning and virtual reality

This study designs and implements an intelligent voice robot music online classroom teaching platform, providing students with a more personalized and interactive learning experience. Research the application of deep learning technology to establish a speech recognition model for music teaching, achieving accurate recognition and understanding of student speech; Combining speech recognition technology and robot interaction technology, construct an intelligent speech robot teaching platform to achieve interactive speech learning; Finally, utilizing virtual reality technology to create an immersive music teaching environment, enhancing students’ sense of participation and learning effectiveness. Through research and analysis of the needs of students and teachers, the functions and characteristics of the intelligent voice robot music online classroom teaching platform have been determined. We have established a comprehensive platform architecture and implemented functional modules such as speech recognition, virtual reality environment construction, and teaching resource management. Designed diverse interactive teaching and learning modes, enhancing the fun and effectiveness of learning. This study successfully designed and implemented an intelligent voice robot music online classroom teaching platform based on deep learning and virtual reality, injecting new vitality and possibilities into music teaching.

The cognitive-emotional dialectics in second language development: speech and co-speech gestures of Chinese learners of English

Abstract This study examines the cognitive-emotional dialectics of Chinese learners of English in terms of both speech and co-speech gestures in their second language development within the lens of Vygotsky’s sociocultural theory. This qualitative study focuses on Chinese learners of English in different learning contexts and analyzes data from interviews and emotional narrative tasks in the English learning experience. Overall findings showed that learners’ emotion and cognition interacted in different learning contexts, and speech and co-speech gestures worked to influence the cognition-emotion dialectics in participant’s perezhivanie. This study implies the inextricable unity of emotion and cognition in second language development, and emphasizes the role of speech and co-speech gestures in shaping second language learners’ perezhivaniya.